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SUMMARY 


\ 

The  problem  of  selecting  the  specific  design  struc- 
ture for  an  operational  test  under  conditions  of  constrained 
sample  size  is  investigated.  The  research  is  limited  to 
univariate,  quantitative,  continuous,  linear  response  models. 

Expected  additional  system  cost  (EASC)  is  defined  and 
modeled  as  the  sum  of  four  general  cost  elements:  fixed  cost 
of  testing;  sampling  cost,  which  is  assumed  to  be  linear; 
expected  cost  due  to  Type  I error;  and  expected  cost  due  to 
Type  II  error.  The  general  form  of  each  variable  element  is 
investigated  as  it  is  affected  by  the  type  of  response  model 
and  by  the  general  design  structure. 

The  problem  of  selecting  the  significance  level,  the 
number  of  observations  for  each  treatment  and  their  alloca- 
tion among  the  cells  within  treatments  with  limited  sample 
sizes  is  formulated  as  a constrained,  nonlinear  optimization 

problem  with  EASC  as  the  objective  function.  The  specific 

io  C P'tVv 

formulation  for  the  2*^  completely  crossed,  fixed  factor  de- 
sign with  a single  covariate  is  developed  and  used  to 
illustrate  the  derivation  of  a solution  procedure  based  on 
a method  of  partial  enumeration.'^  However,  the  procedure  is 
applicable  to  any  factorial  or  ANOCOV  design  so  long  as  the 
response  variable  meets  the  prescribed  limitations.  The 
solution  algorithm  is  programmed  in  FORTRAN  IV.  A listing 


of  this  program  is  included. 

The  approach  is  demonstrated  for  a hypothetical  oper- 
ational test  based  on  a 2^  completely  crossed,  fixed  factor 
design  with  one  covariate. 

The  proposed  approach  is  found  to  be  a valid  method 


for  selecting  a specific  design  structure  for  operational 
testing  within  the  prescribed  limitations,  and  to  be  a use- 
ful decision  making  tool  for  evaluating  design  structure 
decision  alternatives. 
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CHAPTER  I 

INTRODUCTION 


Background 

The  Department  of  the  Army  acquires  new  major  systems 
through  a highly  structured  and  formalized  process  which  is 
prescribed  in  detail  by  a series  of  written  directives  (1, 

2,  3).  The  development  portion  of  this  process  is  divided 
into  three  sequential  phases:  the  conceptual  development 
phase,  the  validation  phase,  and  the  full  scale  development 
phase.  A formal  review  is  conducted  at  the  end  of  each  of 
these  phases  and  the  Secretary  of  Defense  must  give  his  ap- 
proval for  initiation  of  the  next  phase.  Alternatively,  the 
Secretary  of  Defense  may  decide  to  terminate  development,  or 
extend  the  current  phase  for  corrective  action  (9) . In 
making  these  decisions,  he  is  advised  by  the  Defense  Systems 
Acquisition  Review  Council  (DSARC)  which  is  a permanent  body 
tasked  with  reviewing  the  progress  of  all  major  system  devel- 
opments at  the  end  of  each  phase. 

A parallel  advisory  body,  the  Army  Systems  Acquisition 
Review  Council  (ASARC) , exists  at  Department  of  the  Army 
level.  This  body  reviews  all  major  Army  systems  at  the  end 
of  each  development  phase,  and  provides  the  Army's  recom- 
mendations to  the  DSARC  for  incorporation  into  their  recom- 
mendations to  the  Secretary  of  Defense. 
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Both  the  ASARC  and  the  DSARC  rely  heavily  on  the  re- 
sults of  tests  as  the  basis  for  their  recommendations.  These 
tests,  which  are  conducted  during  each  of  the  three  develop- 
ment phases,  are  of  two  distinct  types  with  different  but 
complimentary  objectives.  Developmental  Testing  (DT)  is  con- 
ducted primarily  to  determine  if  the  system  meets  its  tech- 
nical specifications,  and  to  identify  any  need  for  further 
design  and  development.  Operational  Testing  (OT) , on  the 
otherhand,  is  primarily  concerned  with  evaluating  the  opera- 
tional worth  of  the  system  in  terms  of  improved  performance 
capabilities  over  current  standard  or  competitor  develop- 
mental systems.  OT  is  conducted  using  typical  user/operators 
under  conditions  which  duplicate,  as  nearly  as  possible, 
those  under  which  the  system  is  expected  to  perform  if  adopt- 
ed into  the  Army  inventory. 

Normally,  each  development  phase  is  concluded  with 
both  a DT  and  an  OT,  which  precede  the  meeting  of  the  ASARC 
and  DSARC.  Although  DT  and  OT  may  be  conducted  concurrently, 
they  are  required  to  be  evaluated  completely  independently. 
Entirely  separate  organizations  are  responsible  for  each 
type  of  test.  This  research  restricts  itself  to  OT. 

Operational  Testing 

The  US  Army  Operational  Test  and  Evaluation  Agency  is 
tasked  with  the  responsibility  for  planning,  conducting,  and 
evaluating  all  OT  of  major  Army  systems.  Their  objectives 
are  to  evaluate,  under  as  realistic  conditions  as  possible. 
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the  tested  system's; 

1.  Military  utility,  operational  effectiveness,  and 
operational  suitability  (including  reliability,  availability, 
maintainability,  compatability , interoperability,  and  logis- 
tic and  human  engineering  requirements.) 

2.  Desirability,  from  the  user's  viewpoint,  when  com- 
pared to  currently  available  and  competing  developmental  sys- 
tems . 

3.  The  need  for  modifications. 

4.  The  adequacy  of  organization,  doctrine,  operating 
techniques,  and  tactics  for  its  employment,  and  the  system 
for  its  maintenance  support. 

5.  Performance  in  a countermeasures  environment  (1) . 

Operational  testing  is  almost  exclusively  comparative 

in  nature.  Depending  on  the  type  of  system,  one  of  three 
standards  for  comparison  (SFC)  may  be  used:  the  current  stan- 
dard system,  which  the  new  system  is  intended  to  replace;  a 
higher  level  system  (of  which  the  new  system  is  a component) 
without  the  new  system;  or  a set  of  performance  standards 
which  the  new  system  must  meet  or  exceed  (26) . 

The  Operational  Test  and  Evaluation  Agency  (OTEA) 
must  design  each  OT  to  evaluate  the  issue  which  have  been 
prescribed  by  the  Department  of  the  Army.  These  issues  are 
formal  statements  of  the  questions  which  must  be  answered  in 
order  to  determine  the  operational  worth  of  the  new  system. 
The  ]<ey  questions  are  specifically  identified  as  critical 
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issues.  The  list  of  issues  and  their  relative  importance 

I 

may  change  from  one  phase  of  OT  to  another,  based  on  the  re-  | 

I 

suits  of  the  previous  phases.  | 

For  each  issue,  OTEA  develops  an  operational  test  cri- 
terion which  specifies  the  issue  to  be  evaluated,  the  SFC, 
and  the  measures  of  effectiveness  (MOE)  to  be  used  to  eval- 
uate the  issue  (26) . An  MOE  may  be  either  quantitative  or 
qualitative,  depending  on  the  nature  of  the  issue  it  sup-  i 

ports.  A quantitative  MOE  may  be  either  discrete  or  contin- 
uous . 

The  OT  is  then  designed  to  provide  the  data  necessary 
to  compare  the  new  system  with  the  specified  SFC  across  all 
of  the  required  MOE,  with  particular  emphasis  on  those  MOE  ■ 

which  support  critical  issues.  In  addition,  point  or  inter-  ' 

val  estimates  of  the  values  for  each  quantitative  MOE  are  t 

\: 

required  to  determine  the  margin  by  which  the  new  system 
exceeds  or  falls  short  of  the  SFC.  The  main  effects  of  in- 
dependent variables,  other  than  the  treatment  (system)  vari- 
ables are  of  little  interest  in  OT.  Estimates  of  interac-  ji 

tions  between  independent  variables,  other  than  those  in-  |( 

volving  the  treatment  variable,  are  not  required.  Nor  are  j 

OT  intended  to  estimate  functional  relationships  between  de-  Ij 

il 

pendent  and  independent  variables.  This  sort  of  information  i 

is  a function  of  other  types  of  tests. 
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Problem,  Objective,  Scope 

This  research  was  motivated  by  a problem  stated  by 


j 

Hi, 


i 


I 
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OTEA  : 

OTEA  is  continuously  required  to  design  and  analyze  the 
results  of  operational  tests  based  upon  small  sizes 
whether  the  sample  concerns  numbers  of  prototypes,  per- 
sonnel, or  trials.  The  effect  (of  a research  project) 
would  be  directed  at  developing  a methodology  for  de- 
signing, planning,  and  evaluating  operational  tests  of 
limited  sample  size. 

During  OT,  the  new  system  is  in  a prototype  configura- 
tion. Generally,  the  cost  of  an  individual  prototype  of  a 
major  system  dictates  that  only  a small  number  be  construct- 
ed. Due  to  other  requirements,  only  a portion  of  these  pro- 
totypes may  be  available  for  OT,  and  these  may  be  available 
only  for  a limited  amount  of  time.  In  addition,  if  the  new 
system  is  of  a type  which  is  destroyed  in  use  (e.g.  a mis- 
sile) this  imposes  even  greater  restrictions  on  the  number 
of  trials  which  may  be  conducted.  Faced  with  these  restric- 
tions, OTEA  must  design  and  conduct  OT  which  provides  suf- 
ficient data  to  evaluate  all  of  the  required  issues  with  an 
acceptable  precision. 

This  research  addresses  the  first  part  of  the  problem, 
the  design  of  the  OT.  The  objective  of  this  research  is  to 
develop  a methodology  for  selecting  the  design  of  an  OT 
based  on  a criterion  of  minimum  expected  additional  system 
cost.  Recognizing  that  by  selecting  the  design,  the  method 
of  analysis  is  also  determined,  no  designs  are  considered 
for  which  a well  defined  method  of  analysis  does  not  exist. 
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The  limited  statistical  information  required  from  OT 
permits  this  research  to  be  limited  in  two  general  areas: 

1.  Only  linear  response  (MOE)  models  are  considered. 

2.  For  the  factorial  design,  all  factors  are  re- 
stricted to  two  levels. 

These  limitations  are  based  on  the  assumption  that  OT 
are  not  required  to  determine  functional  relationships.  The 
scope  of  this  research  is  further  limited  in  four  additional 
areas : 

1.  Only  two  general  classes  of  designs  are  considered: 
the  completely  crossed,  fixed  factorial  design,  to  include 
fractional  replications;  and  the  analysis  of  covariance 
(ANOCOV)  design.  These  two  classes  of  designs  are  felt  to 

be  sufficient  to  handle  a large  variety  of  OT  situations  and 
to  adaquately  demonstrate  how  the  proposed  methodology  may 
be  applied  to  other  classes  of  designs. 

2.  Only  univariate  response  models  are  considered. 

The  extension  of  the  proposed  methodology  to  the  multivari- 
ate case  is  left  as  a topic  for  future  research. 

3.  Only  continuous,  quantitative  response  variables 
are  considered.  While  this  limits  the  generality  of  the 
method  with  respect  to  OT,  consideration  of  this  requirement 
during  selection  of  MOE  will  minimize  this  limitation.  Fu- 
ture research  may  be  able  to  extend  this  method  to  include 
both  discrete  and  qualitative  MOE. 

4.  The  OT  objective  of  testing  the  hypothesis  of  no 
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treatment  main  effect  is  used  as  the  basis  for  optimization, 
since  this  is  felt  to  be  the  most  ii.portant  information  re- 
quired. However,  modification  of  the  problem  formulation  to 
match  another  hypothesis  as  the  basis  for  optimization  will 
present  no  difficulties  to  an  experimenter  who  is  familar 
with  the  analysis  of  variance  (ANOVA)  and  ANOCOV  methods  and 
who  understands  the  proposed  approach. 

This  research  first  reviews  various  criteria  in  the 
literature  for  selecting  an  experimental  design.  Various 
methods  for  improving  the  precision  of  designs  for  a given 
sample  size  are  also  reviewed,  which  leads  to  consideration 
of  the  two  major  classes  of  designs:  factorial  and  covariate. 
A new  criteria  is  then  developed  based  on  minimizing  the 
expected  additional  system  cost  resulting  from  the  test. 
Expected  additional  system  cost  is  modeled  by  a cost  equa- 
tion consisting  of  four  elements:  fixed  cost  of  experimenta- 
tion; cost  of  sampling;  cost  of  Type  I error;  and  cost  of 
Type  II  error.  The  structure  of  each  element  of  the  model 
is  described  for  the  factorial  and  covariate  designs. 

The  problem  of  limited  sample  sizes  is  then  formu- 
lated as  a constrained,  nonlinear  optimization  problem.  A 
solution  algorithm  is  described  and  demonstrated. 


I 
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CHAPTER  II 

REVIEW  OF  OTHER  APPROACHES  AND 
RELATED  STATISTICAL  TECHNIQUES 

Introduction 

This  chapter  presents  a brief  review  of  the  ap- 
proaches which  have  been  developed  by  previous  researchers 
to  the  problem  of  selecting  experimental  designs  for  the 
linear,  univariate  response  model  experiment.  Methods  of 
improving  the  precision  of  experiments  for  a given  sample 
size  are  also  briefly  reviewed.  The  linear,  univariate  re- 
sponse models  for  ANOVA  and  ANOCOV  are  described,  and  meth- 
ods for  calculating  the  power  of  tests  based  on  these  models 
are  reviewed. 

Approaches  to  the  Selection  of 
Efficient  Experimental  Designs 

Two  general  categories  of  methods  for  measuring  the 
efficiency  of  experimental  designs  were  investigated  during 
this  research.  One  category  consists  of  methods  which 
utilize  a measure  of  the  information  derived  from  a design 
as  an  index  of  its  efficiency.  The  other  category  consists 
of  methods  based  on  the  power  and  significance  level  of  a 
design  with  respect  to  a specific  set  of  hypotheses  to  be 
tested.  With  few  exceptions,  actual  applications  of  both 
methods  are  directed  toward  determining  the  required  number 
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of  observations  and  their  allocation  among  the  experimental 
conditions,  treating  the  class  of  design,  the  number  and 
type  of  independent  variables  and  their  range  of  values  as 
having  been  previously  determined.  With  only  two  exceptions, 
methods  which  actually  sought  to  determine  a specific  optimal 
experimental  design  for  a given  situation  were  limited  to 
those  based  on  some  form  of  information  criterion. 

Information  Based  Approaches 

Using  a measure  of  the  information  produced  by  an 
experimental  design  as  a measure  of  its  efficiency  was  first 
proposed  by  Fisher  (14).  Fisher's  criterion  was  later  re- 
vised by  Cox  (8).  This  criterion  is  defined  by 


= average  variance  (M^  - M^) / (2a^/n) 


I3  = + 3)/(f  + 1) 


(2.1) 

(2.2) 


where  (M.  - M,  ) is  the  difference  between  two  treatment  means, 
D 

2 

f is  the  degrees  of  freedom  for  error,  is  the  variance  of 
the  dependent  variable  for  fixed  values  of  the  independent 
variable,  and  n is  the  sample  size.  Feldt  (11)  utilized  this 
criterion  to  compare  the  relative  efficiency  of  the  two-way 
factorial,  the  analysis  of  covariance,  and  paired  comparison 
designs,  and  to  determine  the  optimum  numbers  of  levels  for 
the  factorial  design  under  various  conditions. 

A majority  of  information  based  approaches  utilize  the 


information  matrix,  S,  defined  as  follows:  given  the  linear 
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model 


E(Y)  = Z 6,  (2.3) 

S = Z (2.4) 

where  Y is  the  response  vector,  6 is  a vector  of  unlcnown  con- 
stants, and  Z is  the  matrix  of  independent  variables  express- 
ed in  standardized  form. 

As  an  example,  Wald  (27)  proposed  a criterion,  W, 
based  on  the  determinant  of  S 

W = (2.5) 

where  the  are  the  eigenvalues  of  S,  and  r is  the  ran)c  of 
S.  Ehrenfeld  (10),  Chernoff  (7),  Kshirsagar  (21),  and  Webb 
(37)  have  proposed  similar  criteria  which  are  functions  of 
the  eigenvalues  of  S. 

More  recently,  Neuhardt,  Bradley  and  Henning  (28) 
have  proposed  the  squared  Euclidean  norm  of  S as  a measure 
of  experimental  efficiency. 

From  the  standpoint  of  OT,  all  of  the  information 
based  criteria  suffer  two  major  disadvantages.  First,  they 
are  difficult  to  interpret  in  terms  which  are  meaningful  to 
the  decision  makers  involved  in  the  systems  acquisition  proc- 
ess. Second,  and  most  important,  since  OT  only  requires 
estimates  of  the  treatment  main  effects  and  interactions,  the 
total  information  produced  by  an  experiment  is  not  a valid 
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measure  of  its  worth. 

Error  Based  Approaches 

Sedransk  (36)  and  Johnson  (20)  are  the  only  examples 
of  the  use  of  error  based  criteria  for  optimizing  experi- 
mental designs  based  on  linear  response  models  which  were 
found  in  the  literature  that  appeared  to  be  applicable  to  OT. 
Sedransk  studied  the  problem  of  allocating  a given  number  of 
samples  among  the  cells  of  2^  factorial  designs  in  order  to 
minimize  sampling  cost  for  a specified  precision.  The  prob- 
lem is  formulated  as  a nonlinear  optimization  problem,  how- 
ever, Sedransk  states  that  standard  nonlinear  programming 
algorithms  are  not  applicable  since  the  objective  function 
does  not  meet  the  convexity  requirements.  Sedransk  suggests 
that  the  problem  may  be  solved  by  a graphical  approach,  or 
by  a systematic  use  of  derivatives.  Johnson,  on  the  other- 
hand,  provides  a general  formulation  for  the  problem  of  de- 
termining the  sample  size,  power,  and  significance  level  for 
a given  design  in  order  to  minimize  total  cost.  However, 
Johnson  carries  the  problem  no  further  than  the  general  prob- 
lem formulation. 

Error  based  cost  criteria  are  directly  applicable  to 
OT  and  present  two  advantages  which  are  the  converse  of  the 
disadvantages  associated  with  information  based  criteria. 
First,  they  are  based  on  cost  versus  risk  which  are  measures 
that  are  directly  applicable  to  decisions  involved  in  the 
acquisition  process.  Second,  they  are  related  to  specific 
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hypotheses  rather  than  to  all  of  the  possible  contrasts  con- 
tained within  a particular  experiment.  Consequently,  they 
may  be  used  to  select  experimental  designs  based  on  their 
criterion  values  with  respect  to  only  those  specific  con- 
trasts (or  estimates)  which  are  of  most  interest  in  OT. 

The  general  ideas  and  problem  formulations  presented 
by  Sedransk  and  Johnson  provided  the  initial  choice  of  di- 
rection for  this  research. 

Methods  for  Improving  Precision 

For  a given  sample  size,  the  precision  of  an  experi- 
ment is  a function  of  the  amount  of  variability  among  the 
observations  which  are  combined  together  to  produce  an  esti- 
mate of  a population  parameter.  Variability  among  observa- 
tions may  be  reduced,  for  a given  sample  size,  in  essen- 
tially three  ways:  by  increased  physical  control  of  the  in- 
dependent variables  which  effect  the  observations;  by  struc- 
tural changes  in  the  experimental  design  which  result  in 
observations  occuring  within  smaller,  more  homogeneous 
groups;  and  by  taking  concurrent  observations  of  the  values 
of  concomitant  variables  which  are  correlated  with  the  re- 
sponse variable,  and  adjusting  parameter  estimates  for  the 
effects  these  variables  are  assumed  to  have  on  the  response 
variable . 

The  entire  field  of  experimental  design  is  either 
directly  or  indirectly  concerned  with  methods  for  reducing 
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this  variability  among  observations.  Only  two  of  several 
basic  design  structures  will  be  reviewed  and  applied  in  this 
research.  These  are  the  factorial  design  and  the  ANOCOV 
design.  Detailed  information  may  be  found  in  any  of  a num- 
ber of  design  of  experiments  texts.  Johnson  (20),  Lindman 
(24),  Scheffe'  (35),  and  Winer  (38)  were  found  to  be  partic- 
ularly useful  for  their  detailed  analysis  procedures  for 
more  complex  designs  including  those  involving  unequal  sam- 
ple sizes. 

Factorial  Designs 

The  factorial  design  structure  seeks  to  reduce  varia- 
bility by  grouping  to  produce  observations  with  as  much 
homogeneity  as  possible  within  groups.  To  the  extent  that 
the  variability  within  groups  is  less  than  the  variability 
between  groups  this  structural  device  results  in  an  increase 
in  precision.  This  condition  will  occur  only  if  the  inde- 
pendent variables  which  are  used  as  a basis  for  grouping  are 
correlated  with  the  dependent  variable  to  be  observed  to  a 
degree  which  will  off-set  the  reduction  in  degrees  of  free- 
dom for  error  which  results  from  grouping.  In  OT  indepen- 
dent variables  are  chosen  because  of  their  expected  correla- 
tion with  the  system  performance,  consequently  this  condi- 
tion should,  in  general,  be  met. 

Each  independent  variable  to  be  considered  during  the 
experiment  is  identified  as  a factor.  The  range  of  values 
for  each  factor  may  be  specified  as  fixed  levels,  or  by 
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fixed  intervals  on  a scale  with  specific  values  for  a par- 
ticular observation  determined  randomly.  All  possible  com- 
binations of  levels  (intervals)  and  factors  may  be  included 
as  experimental  conditions,  yielding  a completely  crossed 
factorial  design  (symbolized  by  AxBxC  for  three  factors) , or 
only  specified  levels  (intervals)  of  one  factor  may  be  al- 
lowed to  occur  with  specified  levels  of  another  factor  yield- 
ing some  form  of  nested  design.  In  addition,  by  assuming 
some  higher  order  interaction  effects  to  be  negligible,  some 
form  of  fractionalized  design  may  be  developed. 

If  all  factors  have  the  same  number  of  levels  (inter- 
vals) , n,  and  there  are  X number  of  factors,  the  design  is 
called  an  factorial  design.  The  completely  crossed, 
fixed  level,  2^  factorial  design  structure  was  selected  for 
this  research  as  being,  in  combination  with  the  ANOCOV  de- 
sign, the  most  generally  applicable  to  OT. 

The  univariate,  linear  response  model  for  a completely 
crossed,  fixed,  2 design  (AxB)  is  commonly  represented  in 
the  literature  as 


= y + a.  + 6.  + ag..  + £. 

ID  ID  idI^ 


(2.6) 


where 


^ijk 


= the  observation  at  the  i^“  level  of  factor 

A and  the  level  of  factor  B, 

= the  population  mean  of  the  observations. 


= the  effect  of  factor  A at  level  i, 

= the  effect  of  factor  B at  level  j , 

= the  interaction  effect  of  factors  A and  B at 
levels  i and  j , 

= the  error  associated  with  the  observation, 


1 .1  .n . .a  . = J^.y,n..B.  = 0 

1]  1 ^1^3  13  3 


(2.7) 


where 


n.  . = the  number  of  observations  in  the  ij  cell. 

11 

The  error  is  assumed  to  be  normally  distributed  with  mean 

o 

zero  and  variance,  a . 

If  only  one  observation  is  taken  from  each  cell,  the 
k subscript  is  omitted.  Under  this  condition,  the  error 
term  is  confounded  v/ith  the  highest  order  interaction  term 
and  cannot  be  estimated  separately.  In  order  to  provide  a 
separate  estimate  of  error,  a minimum  of  two  observations 
per  cell  is  required.  This  constraint  will  be  applied 
throughout  this  research.  Models  for  larger  numbers  of 
factors  are  simply  extensions  of  (2.6). 

ANOVA  is  the  method  of  analysis  associated  with  the 
univariate  factorial  model. 

ANOCOV  Designs 

The  ANOCOV  design  structure  seeks  to  improve  the  pre- 
cision of  an  experiment  through  che  use  of  concomitant  vari- 
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ables.  Such  a variable  is  defined  by  a characteristic  of 
the  response  variable  which  more  or  less  predetermines  the 
general  range  of  its  values  and  is  highly  correlated  with 
it.  Values  of  the  concomitant  variable,  or  covariate,  may 
be  determined  prior  to  the  experiment,  during  the  experi- 
ment at  the  same  time  that  observations  of  the  response 
variable  are  taken,  or  even  after  the  experiment  in  some 
cases.  However,  if  it  is  possible  to  determine  the  values 
before  the  experiment,  strong  consideration  should  be  given 
to  treating  the  concomitant  variable  as  a factor  rather  than 
as  a covariate  in  view  of  the  additional  assumptions  which 
are  required  for  ANOCOV.  Due  to  the  added  assumptions,  use 
of  covariates  should  generally  be  restricted  to  sources  of 
variability  which  cannot  feasibly  be  controlled  by  another 
method.  Such  sources  may  often  occur  in  OT,  consequently, 
this  design  structure  v/as  included  in  this  research. 

The  univariate,  linear  response  model  for  ANOCOV  with 
two  factors  and  a single  covariate  is  commonly  represented 
in  the  literature  as 


Yijk  = ^ 


“i  ^ 


3j  t y(Z. - Z) 


+a6..+E.  (2.8) 

11  Ilk 


where 


Z 


ijk 


the  value  of  the  covariate  for  the  k'*-^  observa- 
tion with  factor  A at  level  i and  factor  B at 
level  j. 
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Z 

7 


the  average  over  all  values  of 

the  linear  effect  of  the  covariate  at  the 


value 


and  all  other  terms  are  as  defined  for  the  factorial  model. 
The  side  conditions  (2.7)  and  the  assumptions  for  the  ANOVA 
model  also  apply  to  ANOCOV.  Four  additional  assumptions 
must  generally  hold  for  ANOCOV: 

1.  Z is  continuous. 

2.  The  population  mean  and  variance  of  Z and  the 
population  value  of  the  correlation  coefficient,  P^Y' 
tween  Z and  Y are  equal  for  all  cells. 

3.  Homoscedasticity  in  Y obtains  around  the  regres- 
sion line  of  Y and  Z. 

4.  Z is  independent  of  all  factors. 

Violations  of  ANOVA  and  ANOCOV  Assumptions 

Much  research  has  been  directed  toward  studying  the 
effects  of  violations  of  the  assumptions  required  by  ANOVA. 

It  has  been  found  that,  in  most  cases,  ANOVA  is  robust  with 
respect  to  all  assumptions  other  than  random  sampling  and 
to  a lesser  degree  homogeneity  of  variance.  Very  little 
appears  to  be  known  about  the  effects  of  violations  of  as- 
sumptions with  respect  to  ANOCOV. 

The  literature  contains  well  known  tests  for  viola- 
tions of  assumptions  which  can  be  applied  during  the  analysis 
of  data  for  both  ANOVA  and  ANOCOV.  Methods  exist  which  will, 
in  many  cases,  reduce  violations  to  an  acceptable  level 
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through  the  application  of  appropriate  transformations  to 
the  data. 

Power  of  the  Variance  Ratio  (F)  Test 
Tlie  Variance  Ratio  or  F test  is  used  for  evaluating 
contrasts  based  on  ANOVA  and  ANOCOV.  The  power  of  the  F 
test  depends  on  the  distribution  of  F when  the  null  hypoth- 
esis is  false.  This  distribution  corresponds  to  Fisher's 
distribution  C (14)  and  is  termed  the  noncentral  F distribu- 
tion, or  F'.  F'  depends  importantly  on  five  factors:  the 

! 

actual  means  of  the  treatment  populations;  the  error  vari- 
ance; the  significance  level;  the  number  of  observations  in 
each  sample;  and  the  degrees  of  freedom  for  error. 

Feldt  and  Mahmond  (12)  have  developed  charts  for 
determining  the  sample  size  required  to  produce  a given  pow- 
er at  a specified  significance  level  for  assumed  values  of 
the  parameter 

(2.9) 

where  I is  the  number  of  treatments  and  and  a are  de- 
fined as  in  (2.6).  These  charts  are  only  available  for  the 
most  commonly  selected  values  of  significance  level.  An- 
other set  of  charts  developed  by  Pearson  and  Hartley  (30) 
determines  the  power  given  the  numerator  and  denominator 
degrees  of  freedom,  the  value  of  (|)  in  (2.9)  and  the  signifi- 
cance level.  Although  more  values  are  charted  than  by  Feldt- 
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Mahmond,  they  still  do  not  provide  a wide  range  of  values 
of  significance  level. 

As  given  by  Patnaik  (29) , the  probability  distribu- 
tion of  F'  is 


2 h ^ 

e 2 


j!B(|v^+j,|v2)\  2 


2Vi+j 


iv^-l+j  /l+v,F' 


"2  (''i'^^2^ 


(2.10) 


where 

v-j^  = numerator  degrees  of  freedom, 

V2  = denominator  degrees  of  freedom, 
X = the  noncentrality  parameter. 

The  power  of  test  is  then  defined  by 


1 - 6 = P (F'  > F_) 

Vi,v2  c 


where  F is  the  critical  value  determined  by  the  given  a, , 
c 1 

and  V2.  Methods  of  evaluating  the  probability  integral 
of  F"  have  been  worked  out  by  Wishart  (39)  . They  involve 
extensive  computational  effort. 

Fortunately,  Patnaik  and  Johnson  have  each  developed 
suitably  accurate  approximations  to  F'  which  may  be  used  to 


calculate  power.  Johnson's  method  was  selected  for  use  in 


2 
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this  research  since  it  is  based  on  a central  F distribution, 
which  is  conceptually  more  understandable  than  Patnaik's 
which  is  based  on  an  incomplete  Beta  function. 

Johnson's  approximation  for  F'  is 


(2.12) 


where  the  noncentrality  parameter,  X,  is  defined  in  general 
by 


X 


EMS 


EMS, 


1 - V. 


(2.13) 


with 

= numerator  degrees  of  freedom  (as  in  F"), 

EMS-]^  = expected  mean  square  of  the  hypothesized  effect 
given  that  the  alternate  hypothesis  is  true, 

EMS^  = appropriate  denominator  expected  mean  square  for 
the  type  effect  to  be  tested. 

For  the  fixed  effects  model,  EMS2  is  simply  the  design  error 

variance,  a^. 

o 

The  Noncentrality  Parameter,  X 

Method  of  Weighted  Squares  of  Means.  In  order  to  cal- 
culate the  value  of  the  noncentrality  parameter,  X,  the  value 
for  EMS^  must  be  determined.  For  the  case  of  equal  sample 
sizes,  n,  this  value  is  given  by  Lindman  (24)  and  others  as 
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EMS^  = + n (2.14) 

However,  for  unequal  sample  sizes  the  computational  formula 
for  EMSj^  becomes  much  more  complex.  Scheffe  (35)  and  Winer 
(38)  both  present  methods  for  calculating  exact  values  of  ! 

EMS^  for  unequal  sample  sizes.  However,  Winer  describes  a ' 

method  which  is  computationally  much  less  complex  for  ap- 
proximating EMS^  using  the  harmonic  mean  of  the  sample  sizes 
for  each  level  of  the  factor  of  interest.  This  technique 
is  called  the  method  of  weighted  squares  of  means. 

X for  the  Factorial  Design.  Under  the  method  of 
weighted  squares  of  means  as  described  by  Winer  the  esti- 
mators of  the  main  effects  for  a two-way  ANOVA  (A x B)  are  { 

: 

I 

= Aj^  - G and  = Bj  - G (2.15)  ! 

where 

h = Bj  = li^ij/a;  G = lJ.AB../ab  (2.16) 

a and  b being  the  number  of  levels  of  factors  A and  B respec- 
tively . 

The  harmonic  mean  of  the  sample  sizes  for  the  i^^ 
level  of  factor  A is  defined  by  Winer  as 


hi  = b/I.(l/n. .) 


(2.17) 
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and  the  sum  of  squares  for  factor  A in  terms  of  as 


ssj  - ‘>[r.h.5.  - 


(2.18) 


Taking  the  expected  value  of  the  SS  and  dividing  by  the 


appropriate  degrees  of  freedom  yields 


EMS,  = + y.h.a."^/a-l 

A O 1 1 


(2.19) 


The  noncentrality  parameter,  X,  may  now  be  calculated 
by  substituting  (2.19)  into  (2.13)  resulting  in 


^ = Iihi<^iVa2 


(2.20) 


X for  the  ANOCOV  Design.  The  expected  mean  square  of 
ANOVA  must  be  adjusted  for  ANOCOV  as  a result  of  the  addi- 
tion of  the  covariate,  Z.  This  results  in  an  adjusted  mean 
square  for  the  effect  of  interest  given  by  the  formula 


EMS*  = (l-p2y)a2  + (2.21) 

2 

where  is  the  design  error  variance  prior  to  addition  of 

the  covariate  and  p is  the  correlation  coefficient  between 

L Y 

Z and  the  response,  Y.  Substituting  into  (2.13)  yields 
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The  Design  Error  Variance,  Oq 


In  order  to  provide  a standard  to  assist  in  evaluat- 
ing alternative  design  structures,  this  research  has  adopted 
a method  used  by  Feldt  (11) . This  method  expresses  the  de- 
sign error  variance  of  the  response  variable  as  a function 
of  the  variance  of  the  response  in  a completely  random  de- 
sign. This  technique  provides  some  useful  insights  into  one 
of  the  principal  effects  that  additional  factors  and  covar- 
iates will  have  on  the  precision  of  the  design,  through 
their  effects  on  the  error  variance.  In  addition,  it  pro- 
vides a means,  through  variation  of  parameters,  of  empiri- 
cally evaluating  the  effects  of  variations  in  the  degree  of 
control  maintained  on  the  levels  of  independent  variables. 

Additional  Factors.  Using  the  standard  assumptions 
for  ANOVA  and  ANOCOV,  Feldt  derived  the  following  formula 


for  the  design  error  variance,  o , of  a two  factor  design 

o 


(Xi  X X-) 


2 2 
'^o  = 


1-0 


(2.23) 


where 


0,.  = 


X.  = 


X^  = 


variance  of  the  response  variable,  Y,  for  a 
completely  random  design, 
the  treatment  factor, 

the  control  factor  (independent  variable) , 
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p = coefficient  of  correlation  between  the  indepen- 

X2Y 

dent  variable  (control  factor),  X2,  and  the  re- 
sponse, Y, 


a = average  variance  of  X about  its  fixed  levels, 

^2 

Ov  population  variance  of  X_. 

/C  ry  b 

For  additional  control  factors,  X^ , •••,  X^,  this 
formula  can  be  repeatedly  applied  to  yield  the  design  error 
variance,  for  n factors 


" '^Y  ^“'^X  Y •••  ^"^X  Y — 2^'\  (2.24) 

L \ ""^2/1  L " \ X/- 


Addition  of  Covariates.  The  change  in  the  design 


error  variance  as  a result  of  adding  a covariate  can  be  ob- 
served by  comparing  (2.20)  and  2.22).  Adding  a covariate  to 

the  n-factor  design  in  (2.24)  yields  a design  error  variance, 
2* 

given  by 


^2*  _ „2' M „2  V 

o o ZY 


(2.25) 


where  p is  the  coefficient  of  correlation  between  the  co- 
zy 

variate,  Z,  and  Y,  the  response  variable. 
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CHAPTER  III 


DEVELOPMENT  OF  CRITERION  AND  APPROACH 


Introduction 

This  approach  defines  expected  additional  system  cost 
(EASC)  as  the  criterion  for  selecting  the  experimental  de- 
sign structure  of  an  OT.  EASC  is  mathematically  modeled  as 
the  sum  of  four  cost  elements.  The  structure  of  each  element 
is  described  with  respect  to  OT  and  methods  for  determining 
cost  coefficients  and  estimating  the  primary  model  parameters 
are  suggested. 

The  design  selection  problem  with  limited  sample  size 
is  formulated  as  a constrained,  nonlinear  optimization  prob- 
lem with  EASC  as  the  objective  function.  A set  of  con- 
straints is  established  based  on  the  characteristics  of  OT. 
The  form  of  the  objective  function  is  investigated  and  an 
algorithm  for  determining  the  optimal  solution  is  described 
based  on  a method  of  partial  enumeration. 


The  Cost  Model 


General  Case 

Expected  additional  system  cost  is  modeled  in  general 
by  the  equation 


EASC  = C^  + - C.  + C a + C„B 

o '-1=1  1 a B 


(3.1) 


where 

= fixed  cost  of  testing, 

N - total  number  of  observations, 

= cost  of  sampling  for  the  i^^  observation, 

Cgj  = penalty  cost  of  a Type  I error, 

Cg  = penalty  cost  of  a Type  II  error, 
a = probability  of  a Type  I error, 

6 = probability  of  a Type  II  error. 

Each  set  of  hypotheses  to  be  tested  has  an  EASC.  Consequent- 
ly, one  set  must  be  selected  as  the  basis  for  optimization. 
The  set  of  hypotheses  with  respect  to  the  treatment  main 
effects  v/as  used  in  this  research.  This  set  of  hypotheses 
may  be  expressed  as 


^o  • ‘'l  ■ ^2  " ^ 


: Pi  - ^2  ^ ° 


(3.2) 


where 

= population  mean  of  the  response  variable  for  the 
i^^  treatment  (i  = 1,2) 

A Type  II  error  is  defined  as  failure  to  reject  the 
null  hypothesis,  when  it  is  false,  i.e.  when  the  alter- 
nate hypothesis,  is  true.  In  order  to  determine  a value 

for  the  probability  of  a Type  II  error,  6,  the  alternate 
hypothesis  of  (3.2)  must  be  stated  in  the  exact  form 


Hi  : - y2 


(3.3) 


27 


\ 


I 


\ 


1 

I 

I 


where  d represents  the  minimum  true  difference  that  will  be 
detected  as  significant  by  the  test.  For  OT,  d represents 
the  specified  performance  margin  by  which  the  new  system 
must  exceed  the  SFC. 

Fixed  Cost  of  Testing.  Fixed  cost  of  testing,  C^, 
includes  all  costs  which  are  independent  of  the  choice  of 
design  structure.  This  includes  the  costs  of  such  items  as 
administrative  and  support  functions,  facilities,  and  the 
minimum  required  test  control  and  data  collection  and  analy- 
sis functions  to  include  personnel  and  equipment. 

Cost  of  Sampling.  This  includes  all  incremental 
costs  associated  with  the  collection  of  an  observation  over 
the  fixed  costs.  This  includes  the  treatment  costs,  and  any 
incremental  costs  required  to  obtain  the  specific  experi- 
mental condition  under  which  the  observation  is  to  be  taken. 
This  research  assumes  that  the  cost  per  sample  will  be  con- 
stant within  treatments,  but  will  vary  between  treatments. 
This  assumption  is  based  on  the  belief  that,  since  OT  are 
generally  conducted  only  on  major  systems,  the  only  signif- 
icant differences  between  sampling  costs  will  be  due  to  dif- 
ferences in  cost  between  the  prototype  test  item  and  the  SFC. 

Errors  and  Penalty  Costs.  Based  on  the  assumed  null 
hypothesis  (3.2),  the  Type  I and  Type  II  errors  which  may 
occur  in  OT  and  their  interpretation  with  respect  to  the  ac- 
tual relative  performance  of  the  tested  system  and  the  deci- 
sions which  are  indicated  by  the  results  of  OT  are  shown  in 


Figure  1.  In  addition,  Figure  1 suggests  a definition  for 
the  penalty  cost  coefficients  for  each  type  of  error.  It 
seems  reasonable  to  use  as  penalties  only  the  additional 
costs  incurred  prior  to  the  next  phase  of  OT,  since  each 
phase  will  reevaluate  the  new  system  and  provide  new  data 
upon  which  to  base  decisions  with  respect  to  its  acceptance, 
rejection,  or  modification. 


Cost  Model  for  the  Factorial  and  ANOCOV  Designs 
For  the  factorial  class  of  designs  equation  (3.1)  be- 


comes 


'1  K 

EASC  = ^ . . . ^ n 


-1=1 


ev=l  -1‘  • '-K  -1’  • --K 


+ Cg6  (ci,X,v^,Vg) 


(3.4) 


where 


K = number  of  factors, 

Li  = number  of  levels  of  the  i^h  factor,  Xj^, 
e.  = level  of  the  i^^  factor. 


£ = number  of  observations  in  the  cell. 


_ = cost  of  an  observation  in  the  e, , 

1 


k 

. th 


, e’“  cell, 
k 


a = significance  level, 

X = noncentrality  parameter  defined  by  (2.13), 
= degrees  of  freedom  between  treatments. 


= degrees  of  freedom  for  error, 


The  form  of  X and  Vp  will  be  determined  by  the  spe- 


I i'  cific  type  of  factors  involved  and  the  pattern  in  which  they 

f : 

1 'V  ! are  to  be  combined.  In  the  case  of  fractionalized  factorial 

I 

f ' designs,  the  defining  contrast  selected  will  determine  the 

cells  from  which  observations  are  to  be  taken.  Also,  any 
higher  order  interactions  which  are  validly  assumed  to  be 

I . insignificant  may  be  pooled  with  the  error  term  to  increase 

; the  error  degrees  of  freedom. 

1 

Addition  of  covariates  does  not  effect  the  form  of 
the  cost  model.  However,  estimates  of  the  sampling  costs 
may  need  revision  to  include  the  additional  cost  per  obser- 
, vation  of  the  covariate.  In  addition,  several  changes  in 

the  primary  parameters  of  the  model  are  required.  The  error 
degrees  of  freedom  must  be  reduced  by  the  number  of  covar- 
iates to  account  for  the  calculation  of  the  average  value  of 
the  covariate.  Also,  the  error  variance  and  expected  mean 
square  of  the  treatment  effect  must  be  adjusted  for  covari- 
ance as  indicated  in  (2.22)  and  (2.25).  The  additional  as- 
sumptions required  for  ANOCOV  must  always  be  considered, 
since  their  violation  could  seriously  effect  the  validity  of 
the  experimental  results. 

Estimation  of  Values  for  Primary  Model  Parameters 

In  addition  to  the  estimates  of  cost  coefficients, 
the  formulation  of  the  cost  model  for  a specific  OT  requires 
estimates  of  the  following  primary  parameters: 

1.  The  error  variance  for  the  response  variable  in  a 
completely  random  design. 

L_Z 
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2.  The  coefficients  of  correlation  between  the  re- 
sponse variable  and  each  factor,  other  than  the  treatment; 
and  between  the  response  variable  and  each  covariate. 

3.  The  ratio  of  the  average  variation  of  each  factor 
about  its  fixed  level  to  its  population  variance. 

These  estimates  are  generally  a problem  only  in  the 
first  phase  of  OT  (OT  I) . In  subsequent  phases  estimates 
will  be  available  from  data  collected  in  previous  phases. 

For  OT  I there  are  several  sources  of  data  which  may  be 
available  to  provide  a basis  for  estimation  of  the  required 
parameters.  If  possible,  a series  of  pre-tests  could  be 
conducted  specifically  designed  to  estimate  these  parameters. 
Alternatively,  data  from  previous  developmental  or  engineer- 
ing design  tests  may  be  used,  or  data  from  previous  tests  of 
similar  type  systems.  Population  variances  of  factors  can 
be  estimated  from  analyses  of  both  combat  and  field  training 
exercise  after-action  reports,  from  studies  conducted  by  the 
Training  and  Doctrine  Command,  from  intelligence  studies, 
and  from  simulations. 


The  Optimization  Problem 


A 2^  completely  crossed  (X,  x X_  x...x  X^)  design  with 

12  K 

all  factors  at  two  fixed  levels  and  a single  covariate,  Z, 


will  be  used  to  illustrate  the  optimization  procedure.  X^ 
will  always  represent  the  treatment  factor.  This  procedure 


may  be  applied  to  any  factorial  or  ANOCOV  design. 
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After  the  cost  model  has  been  determined,  to  include 
estimates  of  all  cost  coefficients  and  primary  parameters, 
the  problem  of  selecting  the  specific  design  structure  for 
an  OT  with  limited  sample  sizes  may  be  formulated  as  a con- 
strained nonlinear  optimization  problem  with  the  cost  model 
as  the  objective  function. 

Assuming  that  (3,2)  and  (3.3)  are  the  hypotheses  se- 
lected as  the  basis  for  optimization,  the  problem  may  be 
stated  as:  find  = {n^j}(i  = 1,2;  j = 1,,..,2^"1)  and  a 
that 

Minimize : 

2 

EASC  = C + I I c.n.  . + C a + B (a , X , v . , v^)  (3.6) 

o i=i  j=i  1 13  a 6 t e 

Subject  to: 


. . < s. 
13-1 

(i  = 

1,2) 

(3.7) 

. . > 2 

13  - 

(for 

all  i and  j) 

(3.8) 

a < 1 

(3.9) 

a > 0 

(3.10) 

All  n^j  integer 


where 
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(3.11) 

(3.12) 


1-p 


X^Y 


(3.13) 


A 


(j2  /h^hA 

ol  \V^2/ 


(3.14) 


,K-1 


h.  = 
1 


,K-1 


I - 1 Vn.  . 
^]=1  ' 1] 


(3.15) 


^ ^ [^(1+A)2/(1+2A) ,v  ^ 1+A  ^c]  (3.16) 

and  S2  represent  the  separate  constraints  imposed 
on  the  maximum  number  of  observations  which  may  be  taken 
with  the  new  system  and  the  SFC,  respectively.  Usually, 

< S2.  The  second  constraint  (3.8)  is  imposed  to  permit  a 
separate  estimate  of  error.  (3.11)  results  from  the  treat- 
ment factor  being  at  two  levels.  (3.12)  is  just  the  formula 
for  the  error  variance  with  unequal  sample  sizes  and  adjusted 
for  the  single  covariate.  (3.13)  results  directly  from  (2.24) 
and  (2.25).  is  the  critical  value  determined  by  the  given 
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a,  and 

Equation  (3.14)  for  the  noncentrality  parameter,  X, 
was  derived  by  solution  of  the  two  simultaneous  equations 


I 


2 

i=l 


h . a . 


1 1 


0, 


(3.17) 


which  is  a necessary  side  condition  for  the  fixed  factor 
model,  and 


“1  - 


(3.18) 


which  comes  directly  from  the  alternate  hypothesis  (3.3) 
since  = y + a^,  for  the  fixed  factor  model.  Solving  (3.17) 
and  (3.18)  for  the  a^'s  yields 


a-,  = 


hj+h2 


Cl^  = 


hith2 


(3.19) 


Substituting  (3.19)  into  the  formula  for  X for  the  three-way 
design  results  in  (3.14).  Note  that  the  value  of  d must  be 
expressed  in  the  same  units  as  the  response  variable. 

Method  of  Solution 

The  algorithm  proposed  by  this  research  for  solving 
the  optimization  problem  as  formulated  in  (3.6)  through 
(3.10)  was  motivated  by  the  partial  enumeration  method  de- 
scribed by  Neuhardt  and  Bradley  (27)  . However,  the  two  meth- 
ods differ  both  in  the  form  of  the  objective  function  and  in 
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procedure . 

This  method  utilizes  a sequential  analysis  of  the 
functional  relationships  between  the  constants,  parameters, 
and  variables  of  the  objective  function  and  constraints  to 
eliminate  within  the  set  of  feasible  solutions  from  consid- 
eration as  potentially  optimal  resulting  in  what  is  termed 
a "reduced  feasible  set"  which  is  small  enough  to  make  a 
total  enumeration  over  the  remaining  elements  computation- 
ally feasible  using  a digital  computer  to  perform  the  re- 
quired calculations.  Without  the  constraint  of  small  sample 
sizes,  this  method  is  neither  computationally  feasible  nor 
operationally  required. 

Derivation  of  the  method  of  solution  is  described  by 
the  following  steps: 

1.  Setting  a lower  bound  of  two  observations  per  cell 
eliminates  from  feasibility  all  combinations  of  observations 
not  meeting  this  constraint. 

2.  It  is  a well  known  fact  in  the  literature  that  6, 
the  probability  of  a Type  II  error,  varies  inversely  with 
the  noncentrality  parameter,  X,  This  fact  is  illustrated  in 
the  charts  by  Pearson  and  Hartley. 

3.  From  (3.14)  it  can  be  seen  that  A varies  directly 
with  the  hj^'s,  since  h^  ^ 1, 

4.  From  (3.15)  it  may  be  shown  that,  for  a given 

total  number  of  samples  for  the  i^^  treatment,  T^ , such  that 
^jUfj  ~ "^i ' maximum  value  for  h^  will  occur  when  all  of 
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the  associated  n^^'s  are  as  nearly  equal  as  possible,  con- 
sidering that  they  are  integers. 

5.  From  the  relationships  described  in  steps  1,  2 
and  3,  the  minimum  value  for  6 for  given  and  a will  occur 
when  all  of  the  n^^'s  within  each  treatment  are  as  nearly 
equal  as  possible. 

-6.  From  step  5,  if  all  samples  within  a given  treat- 
ment level  have  the  same  cost,  Cj^,  which  is  assumed  to  be 
the  case  for  most  OT,  the  logical  choice  of  allocation  of 
the  total  number  of  samples  will  be  the  most  even  allocation 
possible,  since  the  distribution  of  a given  number  of  sam- 
ples among  cells  has  no  effect  on  EASC  except  through  its 
effect  on  6.  This  reduces  the  feasible  set  by  removing  all 
distribution  patterns  which  do  not  meet  this  condition.  It 
is  important  to  note  that  if  the  sample  costs  per  cell  are 
not  equal,  this  choice  could  not  be  so  easily  determined. 
Since,  the  increase  in  cost  resulting  from  increasing  6 by  a 
more  imbalanced  allocation  of  samples  may  be  more  than  off- 
set by  a decrease  in  sampling  cost  resulting  from  taking 
fewer  samples  from  high  cost  cells  and  more  samples  from  low 
cost  cells. 

7.  Since  the  allocation  of  observations  to  specific 
cells  will  be  accomplished  by  a random  process,  after  the 
optimal  number  of  samples  and  their  distribution  pattern  is 
determined  for  each  row,  for  purposes  of  determining  the 
optimal  distribution  of  samples  the  artifical  constraint. 


^ permutations  of  a specific  distribution  pattern  from  the 


■ feasible  set.  In  otherwords , as  far  as  the  optimal  solution 

‘ is  concerned,  the  order  in  which  observations  are  distrib- 

r 

' < 

uted  among  the  individual  cells  is  immaterial  since  for 
given  i,  any  ordering  of  a given  set  of  n^^j's  will  produce 
the  same  value  of  the  objective  function,  all  other  condi- 
. tions  being  fixed- 

8.  From  (2)  thru  (7),  for  any  given  there 

will  be  a unique  within  the  reduced  feasible  set. 

From  (3.7)  and  (3.8)  , 2^  _<  S^. 

9.  For  any  given  total  number  of  observations,  N, 
there  is  only  a finite  set  {(T-|L,T2)  |T-|^  + T2  = N}. 

10.  If  the  cost  of  sampling  for  the  new  system,  Cj^, 

is  greater  than  the  cost  for  the  SFC,  C2f  which  is  generally 
true  for  OT,  then  no  solution  in  which  ^ will  be 
optimal,  since  transposing  the  values  of  and  produces 
the  same  value  for  B at  a reduction  in  sampling  cost,  all 
other  conditions  being  fixed.  Therefore,  for  given  N,  the 
reduced  feasible  set  now  becomes  {(Tj^,T2)  | T^  + = N,  T^<T2}- 

11.  Combining  (8),  (9),  and  (10),  for  given  a and  N, 
the  reduced  feasible  set,  (s|a,N),  may  be  defined  by 

(slct,N)  = {(n  ,n,)  ILn  = T.  , 5;.T  = N,  T < T } (3.20) 

jl^  1 11  1 ^ 


where 
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j 

Hi  = ^nij|nij>  2,  rii^  > ni|^(k>j)}  (3.21) 

I 

12.  Enumeration  over  (S|a,N)  will  identify  the  optimal  | 

13.  Enumeration  over  (s(a,N)  for  all  possible  values 

of  N (2  _<  N _<  + S2)  will  identify  the  optimal  (ni,n2|a). 

14.  Enumeration  over  (s|a,N)  for  all  possible  values 
of  N and  incrementally  increasing  values  of  a,  starting  with 
a at  a lower  bound  close  to  zero,  will  identify  the  overall 
optimal  (ni,n2,a).  The  lower  bound  and  increment  for  a must 
be  selected  by  the  test  designer.  A lov/er  bound  of  .01  is 
suggested.  Initially,  a relatively  large  increment,  say  .1, 
may  be  used  to  determine  the  values  for  a between  which  the 
optimum  point  lies.  Then  a search  between  these  values 
using  a smaller  increment  may  be  used  to  more  closely  ap- 
proach the  true  optimum.  One  caution  must  be  observed  in 
this  approach:  if  the  initial  increment  is  chosen  too  large, 
the  minimum  point  for  EASC  may  not  be  detected.  To  minimize 
this  problem  it  is  recommended  that  the  initial  increment  be 
chosen  as  small  as  possible  while  still  limiting  the  points 
to  be  evaluated  to  a computationally  feasible  number. 

15.  In  evaluating  the  (nj^,h2|a,N)  the  procedure  is  to 
start  with  the  minimum  value  of  N and  the  minimum  value  of  T^^ 
for  the  given  N which  is  the  point  yielding  the  minimum  sam- 
pling cost  and  the  maximum  6 cost,  since  \ is  a minimum.  For 
given  N,  increasing  T^^  results  in  an  equal  decrease  in  T2, 
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call  it  AT.  It  also  results  in  an  increase  in  X,  AX,  which 
produces  a decrease  in  3 cost,  ABC;  and  an  increase  in  sam- 
pling cost,  ASC.  The  problem  is  to  determine  the  maximum 
T^  for  which 


AT 


(3.22) 


where  as  AT  increases,  AX/AT  decreases,  but  is  always  posi- 
tive. Also,  as  X increases,  ABC/AX  decreases  continuously, 
reaches  a minimum,  and  then  increases,  but  is  always  nega- 
tive. This  minimum  point  may  not  occur  within  the  range  of 
X for  a given  N.  ASC/AT  =0^-02,  is  constant. 

The  form  of  the  non-central  F probability  density 
function  precludes  taking  partial  first  and  second  deriva- 
tives in  order  to  establish  the  exact  form  of  the  objective 
function  as  a function  of  (T^,T2la,N),  however,  an  empirical 
analysis  indicates  that  it  possesses  a single  minimum  which 
is  located  either  at  an  interior  point  or  at  one  of  the  end 
points  of  the  range  defined  for  (T^^  ,T2  1 a ,N)  . In  the  next 
section  the  basis  for  this  conclusion  is  presented  in  greater 
detail. 


16.  Based  on  the  conclusion  in  the  preceding  para- 
graph, if  a point  (Tj^,T2|a,N)  is  reached  as  T^  increases  for 
which  EASC  is  a local  minimum,  then  either  this  point  or  the 
end  point  where  T^  = "^Imin  given  N is  optimal  for  the 


40 


I 


i 


given  values  of  a and  N,  and  no  longer  values  of  need  be 
evaluated  for  that  a and  N. 

17.  The  same  rationale  described  for  AEASC/AT^  in 
the  preceding  paragraphs  applies  to  AEASC/AN,  for  optimal 
(Ti,T2|a,N).  This  is  true  since  choosing  the  optimal 
(Ti,T2(a,N)  identifies  the  minimum  point  on  the  segment  of 
the  EASC  curve  for  given  N and  a described  in  the  following 
section  and  illustrated  in  Figure  5.  Consequently,  if  a 
point  (T^,T2|a,N)  is  reached  as  N increases  for  which  EASC 
as  a function  of  N for  optimal  (T^,T2la,N)  is  a local  mini- 
mum, then  either  this  point  or  the  end  point,  optimal 

(Ti  ,T2  I a ,Nj^in)  , is  optimal  for  the  given  value  of  a,  and  no 
larger  values  of  N need  be  evaluated  for  that  value  of  a. 

18.  As  indicated  in  step  14,  the  procedure  for  eval- 
uating the  {n^,n2|a,N)  is  to  start  with  a at  a lower  bound 
close  to  zero.  It  is  a well  known  fact  that,  all  other 
parameters  being  fixed,  as  a increases,  6 decreases  and  ap- 
proaches zero  as  a approaches  one,  its  maximum  value.  There- 
fore, for  incrementally  (Aa)  increasing  values  of  a,  there 
will  be  at  most  one  point  where  for  optimal  (nj^,n2fN|a) 


A EASC 
Aa 


(3.23) 


In  other  words,  the  objective  function,  EASC,  as  a function 
of  a for  optimal  (nj^,n2,N|a)  possesses  a single  minimum 
point  which  is  located  either  at  an  interior  point  or  at  one 


A 


I 
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of  the  end  points  of  the  range  of  a.  This  relationship  is 
empirically  demonstrated  by  Figures  10  thru  13. 

19.  Based  on  the  relationship  described  in  step  17, 
if,  for  increasing  a,  a value  of  a is  reached,  call  it  a^, 
where  EASC  for  optimal  (n^,n2»N|a^)  > EASC  for  optimal 

,n2 /N I a^-Aa)  , then  the  optimal  value  for  a,  call  it  a*, 
lies  in  the  interval:  {a^-2Aa)  _<  a*  < a^.  No  larger  values 
of  a need  be  evaluated. 

The  EASC  Algorithm 

The  EASC  algorithm  based  on  the  derivation  described 
in  the  preceding  section  consists  of  the  following  step-by- 
step  procedure. 

1.  Obtain  the  required  input  data. 

a.  Design  structure  parameters. 

(1)  Number  of  factors,  K. 

(2)  Number  of  covariates,  M. 

b.  Sample  size  constraints. 

c.  Cost  coefficients. 

(1)  Alpha  cost  coefficient,  C^ . 

(2)  Beta  cost  coefficient,  C„ . 

p 

(3)  Treatment  sample  costs,  Cj^  and  C2. 

d.  Primary  parameters. 

2 

(1)  Random  design  error  variance, Oy. 

(2)  Correlation  coefficients  for  control 


factors,  p y,  . . . / P j.  y- 
2 K 


V- 


*Z 


(3)  Correlation  coefficients  for  covariates , 


(4)  Average  control  variances  for  control 


factors,  . 


(5)  Population  variances  for  control  factors, 


2 2 

O /•••/O  • 


e.  Initial  value  of  a,  a^. 

f.  Increment  for  a,  Aa. 

2.  Calculate  the  value  for  the  design  error  variance, 

2 

a , by  (3.13)  and  for  the  error  degrees  of  freedom,  v , by 
o ® 

(3.12) . 

3.  Calculate  lower  and  upper  bounds  for  total  number 
of  observations,  N as  follows: 

a.  H . . 


>>•  "max  = =1  + 

4.  Starting  with  a = a^,  do  the  following  to  deter- 


mine the  optimal  (n^,n2fN|a): 


a.  Starting  with  N = do  the  following  to 

determine  the  optimal  (n^,n2lN,a): 


(1)  Determine  the  initial  values  of  (T^,T2|n) 


as  follows: 


(a)  If  N - S2  < 2^,  = 2^,  other- 


'^Imin  = N - S2. 


T2  = N - 
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(2)  Determine  the  which  results  in 

the  most  even  allocation  possible,  considering  that  the 
n^j's  are  integers,  and  where,  n^  = > 2, 

^ij  ^ ^ik  follows: 

(a)  For  i = 1,2  and  j = 1,...,2^  let 
n^j  = (Tj^/2^~^)  - (2^“^Modulo  T^)  . 

(b)  For  i = 1,2  and  j = 1,..., 

(2^  ^Modulo  T- ) : add  1 to  n. . . 

1 ID 

(3)  Calculate  the  (hj^|n^)  by  (3.15). 

(4)  Calculate  the  (A|hj^,h2)  by  (3.14). 

(5)  Determine  (F  la,v  ,v  ) by  using  tables 

c t ^ 

or,  if  using  a computer  program,  by  a prepackaged  subrou- 
tine . 

(6)  Determine  (3|X,F  ,v  ) from  (3.16)  by 
interpolation  from  tables  or  by  computer  subroutine. 

(7)  Calculate  (EASC  n^^  ,n2  ,N,a  , B)  from  (3.6) 
and  record  this  value. 

(8)  If  T,  = T,  . , set  the  minimum  value  for 

1 Imin 

(EASC  N,a)  equal  to  the  value  for  EASC  just  calculated  in 
(7) , record  this  value  and  skip  to  (10) , otherwise,  continue 
with  (9)  . 

(9)  Compare  the  value  of  EASC  just  calculated 
in  (7)  with  the  current  minimum  value  of  (EASC  N,a).  If  it 
is  less,  make  it  the  new  minimum  value  of  (EASC  N,a),  record 
this  value,  and  continue  with  (10) , otherwise,  check  to  de- 
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termine  if  EASC  for  the  previous  value  of  (T^,T2)  is  a local 
minimum.  If  it  is,  skip  to  (11)  , otherwise  continue  with 
(10)  . 

(10)  If  and  T2  > 2^"^,  add  one  to 

and  subtract  one  from  T2  then  return  to  step  4a  (2),  other- 
wise, continue  with  (11) . 

(11)  Set  optimal  (EASC  N,a)  equal  to  the  cur- 
rent minimum  value  of  (EASC|N,a),  record  this  value  and  con- 
tinue with  4b. 

b.  If  N = set  the  minimum  value  of  (EASC|a) 

equal  to  the  current  (EASC|N,a),  record  this  value  and  skip 
to  4d,  otherwise,  continue  with  4c. 

c.  Compare  the  optimal  (EASC|N,a)  just  deter- 
mined in  4a(ll)  with  the  current  minimum  value  of  (EA5C|a). 

If  it  is  less,  make  it  the  new  minimum  value  of  (EASC | a)  and 
continue  with  4d , otherwise,  check  to  determine  if  optimal 
EASC  for  the  previous  value  of  N is  a local  minimum.  If  it 
is,  skip  to  4e,  otherwise,  continue  with  4d. 

d.  If  N < increase  N by  one  and  return  to 

step  4a  (1),  otherwise  continue  with  4e. 

e.  Set  optimal  (EASC | a)  equal  to  the  current  min- 
imum value  of  (EASC I a),  record  this  value  and  continue  with 
5. 


5.  If  a = a^,  set  overall  minimum  value  of  EASC  equal 
to  the  current  optimal  (EASC | a),  record  this  value  and  skip 


^ 

J 

45  I 


i 


k 


6.  Compare  the  optimal  (EASC|a)  just  determined  in 
4e  with  the  current  overall  minimum  value  of  EASC.  If  it 
is  less,  make  it  the  new  overall  minimum  value  of  EASC  and 
continue  with  7,  otherwise,  skip  to  8. 

7.  If  a < 1.0  - Aa,  increase  a by  Aa  and  return  to 
step  4a,  otherwise,  continue  with  8. 

8.  Set  the  overall  optimal  EASC  equal  to  the  current 
minimum  value  of  EASC.  This  is  the  last  step  of  the  algo- 
rithm. By  recording  the  values  of  T^,T2,N  and  a for  each 
value  of  EASC,  the  algorithm  yields  the  overall  optimal 
{n2^,n2/0t)  • 

As  previously  stated,  the  actual  assignment  of  obser- 
vations to  specific  cells  and  the  sequence  of  observations 
must  be  determined  by  a random  process  constrained  to  the 
optimal  distribution  pattern  determined  by  the  EASC  algo- 
rithm. 

This  algorithm  was  programmed  in  FORTRAN  IV  for  the 
Georgia  Institute  of  Technology's  CDC  CYBER  70  computer.  A 
computer  listing  of  this  program  and  description  of  the  out- 
put options  is  contained  in  the  Appendices. 

Empirical  Analysis  of  Objective  Function 

The  computer  programmed  solution  algorithm  was  used 
to  generate  data  for  a 2^  completely  crossed  design  with  one 


covariate  based  on  hypothetical  values  of  the  cost  coeffi- 
cients and  the  primary  parameters  in  order  to  test  the  pro- 
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gram  and  empirically  investigate  the  functional  relation- 
ships between  the  objective  function  and  the  decision  vari- 
ables, n^.n^fN,  and  a.  With  the  exception  of  Figure  4,  all 
illustrations  in  this  section  are  based  on  this  data.  The 
values  for  cost  coefficients  and  primary  parameters  not  in- 
dicated on  the  figures  themselves  are  contained  in  Appendix 
B. 

EASC  as  a Function  of  ,T2  | a ,N) 

Each  (T^,T2(a,N)e  S,  defines  a specific  value  for  the 
noncentrality  parameter,  A,  which  increases  monotonically 
with  as  illustrated  in  Figure  2.  The  values  of  3 as  a 
function  of  X derived  from  the  test  data  are  shown  in  Figure 
3.  The  variations  in  A for  given  are  caused  by  the  dif- 
ferent values  for  (T, ,T_|v  ).  Consequently,  3 cost,  as  a 

-1-  2 e 

function  of  T^,  for  continuous  T^,  is  of  the  general  form 
shown  in  Figure  4 . 

The  form  of  the  derivative  of  3 cost  with  respect  to 
Tj^ , 3 3C/3Tj^,  is  also  shown  in  Figure  4.  The  partial  deriva- 
tive of  sampling  cost  with  respect  to  T^,3SC/3Tj^,  is  a con- 
stant (c-j^  - C2)  • Note  since  Cq^  > C2  as  discussed  in  step  10 
of  the  previous  section,  3SC/3T^>  0.  However,  to  more 

clearly  show  the  relationships  between  33C/3T^,  Figure  4 
illustrates  the  negative  of  3SC/3T^.  For  continuous  T^ 


lim  33C 

Tj^-*-0  3Tj^ 


lim  3 3C 
T,->”  3T, 


0 and 


0 


(3.24) 


Figure  4.  EASC  as  a Function  of  (T  ,T 


a,N)  - General  Forn 


Based  on  these  relationships,  EASC  as  a function  of  (Tj^, 
T^lajN)  will  have  one  of  the  three  general  forms  shown,  de- 
pending on  the  relationship  between  the  minimum  value  of 
9SC/3Tj^  and  -9SC/8Tj^ 

min  9 3C/9T^  < -9SC/9T^  =i>  EASC  (1) 

min  9 6C/9Tj^  = -9SC/9T^  =>  EASC  (2) 

min  96C/9T^  > -9SC/9T^  =>  EASC  (3) 

However,  since  is  bounded  from  above  and  below,  only  a 
segment  of  the  curves  shown  in  Figure  4 will  actually  occur. 
Also,  since  is  integer,  only  discrete  points  within  that 
segment  will  be  generated.  Figure  5 illustrates  these  seg- 
ment of  the  EASC  curve  which  were  obtained  from  the  example 
data  for  several  different  values  of  N.  Note  how  increasing 
the  value  of  N shifts  the  segment  of  the  EASC  curve  which 
actually  occurs  from  right  to  left  with  respect  to  Figure  4. 

Figure  6 illustrates  the  effect  of  increasing  the  sig- 
nificance level,  a.  Note  that  as  a increases  the  segment  of 
the  EASC  curve  which  occurs  appears  to  shift  from  left  to 
right  with  respect  to  Figure  4.  What  is  actually  occuring 
is  that  the  increasing  a causes  all  of  the  curves  in  Figure 
4 to  be  compressed  to  the  left.  This  is  due  to  the  fact  that 
as  a increases  for  given  N the  rate  of  change  of  6 with  re- 
spect to  increases.  Note  also  in  Figure  6 that  increasing 
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Figure 


6.  EASC  as  a Function  of  (T  ,T  |a,N)  - Computed 
Example.  ^ ^ 
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a causes  wach  value  of  EASC  to  be  increased  by  a constant, 

C Aa. 
a 

EASC  as  a Function  of  N for  Optimal  (n^^ , n2  | « » N) 

Selecting  for  each  value  of  N the  optimal  allocation 
of  observations,  (h^,h2)/  results  in  the  EASC  values  shown 
in  the  EASC  values  shown  in  Figure  7.  Note  that  as  the  sig- 
nificance level  increases,  the  optimal  number  of  observations 
initially  increases,  then  decreases.  This  is  the  result  of 
the  variations  in  the  rate  of  change  of  6 with  respect  to  N 
for  given  values  of  a and  N.  Where  this  rate  is  high  enough 
to  off-set  the  increase  in  sampling  cost,  increasing  N will 
reduce  EASC.  Once  this  rate  decreases  to  the  point  where 


„ ^ < ASC 

^ AN  AN 


(3.25) 


then  increasing  N will  increase  EASC.  Figures  8 and  9 show 
this  relationship  and  how  it  is  effected  by  the  magnitude  of 
C and  by  the  ratio 


r = d^/a^  (3.26) 

o 

Note  that  as  r increases  (r  = .022  to  .086),  initially  the 
optimal  (T^,T2fN|a)  increases,  but  as  r continues  to  increase 
(r  = .086  to  8.67),  it  decreases.  This  is  due  to  the  effect 
that  variations  in  r have  on  the  rate  of  change  of  6 with 
respect  to  N for  given  a.  Initially,  as  r increases,  it  pro- 
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Figure  7.  EASC  as  a Function  of  N for  Optimal  (T, ,T- 
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duces  large  changes  in  6 with  respect  to  N,  however,  as  it 
continues  to  increase,  6 rapidly  approaches  zero  for  given  a, 
consequently,  the  condition  in  (3.24)  occurs  at  increasingly 
smaller  values  of  (Njo).  The  effect  of  r on  6 is  indirect 
through  its  effect  on  X.  This  can  be  seen  by  rewriting 
(3.14)  in  terms  of  r 

X = 4 r r + 3 (3.27) 

hi+h2 

The  effect  of  the  value  for  C on  the  optimal  value  of  N is 

P 

also  through  its  effect  in  (3.24). 

EASC  as  a Function  of  a for  Optimal  (nj^,n2,N|a) 

Figure  10  shows  the  optimal  value  of  the  objective 

function,  EASC,  for  given  a.  This  figure  clearly  shows  that 

the  optimal  value  of  (EASC | a)  is  sensitive  to  variations  in 

r.  This  sensitivity  increases  rapidly  as  the  ratio  C./C 

p a 

increases.  Also,  the  optimal  value  of  a becomes  increasingly 
sensitive  to  variations  in  r as  C„/C^  increases.  These  ef- 

P Ot 

fects  are  directly  a result  of  the  relationships  described 
in  the  preceding  section.  From  the  standpoint  of  OT,  this 
illustrates  that,  in  general,  the  larger  the  required  per- 
formance margin,  the  easier  the  experimenter's  job  will  be. 

Summary  of  Procedure 

The  basic  procedure  for  the  design  of  an  OT  described 
during  this  research  is  briefly  summarized  as: 


I 
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1.  Determine  minimum  number  and  type  of  factors  to 
be  considered  and  how  they  are  to  be  combined  to  determine 
the  conditions  under  which  observations  will  be  taken.  The 
minimum  number  of  factors  will  generally  be  dictated  by  the 
test  issues. 

2.  Determine  response  variable  to  be  measured  (MOE) . 
This  must  be  a continuous  variable. 

3.  Formulate  the  appropriate  response  model  based  on 
Steps  1 and  2. 

4.  Select  the  set  of  exact  hypotheses  to  be  used  as 
the  basis  for  optimizatation.  Normally,  this  will  be  the 
null  hypothesis  of  no  treatment  effect  versus  an  exact  form 
of  the  alternate  hypothesis:  the  tested  system  exceeds  the 
SFC  by  the  required  performance  margin. 

5.  Determine  the  cost  model  to  include  estimates  of 
all  cost  coefficients  and  primary  parameters. 

6.  Formulate  the  optimization  problem  to  include  all 
constraints . 

7.  Apply  the  EASC  algorithm  to  determine  the  number 
of  observations  to  be  taken  in  each  row  and  their  distribu- 
tion, the  level  of  significance,  and  the  power  of  the  test. 

8.  Use  a random  process  to  assign  observations  to 
specific  cells  and  to  determine  the  sequence  in  which  obser- 
vations are  to  be  taken. 


9.  Vary  the  control  limits  on  the  levels  of  factors 
to  determine  the  optimum  control  required  if  control  is  an- 
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ticipated  to  become  a problem. 

10.  Repeat  Steps  5,  6,  and  7 for  any  alternatives 
which  may  be  of  interest  to  the  experimenter  such  as  addi- 
tion of  a blocking  factor  or  covariate;  an  increase  in  the 
number  of  observations,  if  the  previous  optimal  solution  oc- 
curred at  the  upper  limit  of  this  constraint  for  one  or  both 
treatments;  or  fractional  replication. 

11.  Select  the  optimal  feasible  alternative. 

12.  Begin  experimentation. 

13.  Correct  estimates  of  input  parameters  as  test  data 
becomes  available. 

14.  Repeat  Step  7 and  other  steps  as  necessary  to 
determine  the  effect,  if  any,  of  the  corrected  parameter 
estimates  on  the  optimal  solution. 


•V. 

':f'i 
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CHAPTER  IV 

DEMONSTRATION  OF  THE  APPROACH 

Introduction 

This  chapter  presents  a brief  demonstration  of  the 
approach  developed  in  Chapter  III.  This  demonstration  •>  Mus- 
trates  the  iterative  use  of  this  methodology  as  more  accurate 
estimates  of  input  parameters  become  available,  cost  coeffi- 
cients change,  and/or  new  alternatives  become  available  to 
the  test  designer.  Variation  of  the  input  parameters  and 
cost  coefficients  permits  the  evaluation  of  any  number  of 
alternatives.  This  demonstration  includes  only  a few  of 
these  as  illustrations.  A hypothetical  OT  reguirement  is 
used  as  a basis  for  this  demonstration. 

The  Requirement 

The  Commander,  US  Army  Operational  Test  and  Evalua- 
tion Agency  (OTEA)  has  been  directed  to  conduct  operational 
tests  to  evaluate  the  overall  military  worth  of  the  new 
ground-to-air  tactical  missile  system,  TAAMS,  which  is  under 
development  as  a replacement  for  the  current  standard  HAWK 
missile  system. 

The  plan  of  test  calls  for  separate  sub-tests  of  the 
major  subsystems  of  TAAMS.  One  of  these  subsystems  is  the 
missile  guidance  system.  The  plan  of  test  states  that. 


62 


t • 

r 


I 

i 

since  the  missile  warhead  is  detonated  by  the  proximity  of  I 

the  missile  to  the  target  aircraft,  the  most  critical  issue 
for  evaluation  of  the  guidance  system  is  its  accuracy.  The  i 

ambient  temperature,  altitude  of  the  target,  and  speed  of  I 

the  target  are  indicated  as  the  most  likely  factors  to  have 
a significant  effect  on  the  accuracy  of  the  guidance  system.  | 

The  project  manager  for  TAAMS  has  notified  OTEA  that 
a maximum  of  12  missiles  may  be  fired  in  each  phase  of  OT  to  ‘ 

evaluate  the  guidance  system.  In  addition,  the  Department 
of  the  Army  has  specified  that  no  more  than  20  HAWK  missiles 
may  be  fired  during  each  phase  of  OT  for  evaluation  of  the 

1 

guidance  system. 

For  the  guidance  system  sub- test.  Commander,  OTEA  has 
directed  that  a comparative  test  be  conducted  using  live 
firings  of  the  two  missile  systems  against  drone  aircraft 
targets.  He  further  specified  that  the  standard  for  compar- 
ison (SFC)  will  be  the  HAWK  missile  and  guidance  system,  and 
that  the  measure  of  effectiveness  (MOE)  for  accuracy  of  the 
guidance  systems  will  be  the  mean  miss  distance  from  the 
target  as  measured  by  a radar  mounted  on  the  drone  and  re- 
corded by  telemetry. 

Although  point  and  interval  estimates  of  the  values 
of  the  MOE  are  required,  as  well  as  tests  for  interaction 
effects,  the  principal  purpose  of  the  sub-test  is  to  deter- 
mine if  the  accuracy  of  TAAMS  exceeds  the  HAWK  by  the  re- 
quired performance  margin.  Consequently,  the  following  hy- 
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potheses  were  chosen  as  the  basis  for  selecting  an  optimal 
design  through  use  of  the  EASC  approach: 


versus 


I 


% : Wt  ■ 

where  T indicates  TAAMS  and  H indicates  HAWK. 


Test  Design  for  OT  I 

After  evaluating  the  test  directive,  the  guidance  sys- 

3 

tern  sub-test  designer  recommended  a 2 completely  crossed 
factorial  design  with  ambient  temperature,  Z,  to  be  treated 
as  a covariate,  and  recorded  at  the  launch  site  just  prior 
to  each  firing.  The  two  independent  variables  to  be  treated 
as  control  factors  are:  speed  of  target,  and  altitude  of 
target,  X2.  Ambient  temperature  was  selected  as  a covariate 
because  of  the  difficulty  in  maintaining  any  control  over  it. 
Also,  the  test  designer  feels  that  it  will  sufficiently  meet 
the  ANOCOV  assumptions. 

The  test  designer  now  wishes  to  determine  the  number 

of  firings  which  should  be  used  for  each  type  of  m.vjsile, 

the  distribution  of  these  observations  among  the  cells  of 
3 

the  2 design  the  level  of  significance  to  be  set  for  the 
test,  and  the  resultant  power  of  the  test  with  respect  to 
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cost  coefficients  and  primary  parameters  required  for  the 
EASC  approach.  These  estimates  are  shown  in  Table  1.  He 
further  determines  that,  since  tests  of  interaction  effects 
are  required,  a minimum  of  two  observations  will  be  allo- 
cated to  each  design  cell. 

Inputing  the  data  contained  in  Table  1 and  the  sample 
size  constraints  imposed  by  the  project  manager  and  the 
Department  of  the  Army  into  the  EASC  program  produced  the 
values  shown  in  Figure  11.  Note  that  the  values  for  EASC 
beyond  the  optimal  value  of  a are  shown  throughout  this  dem- 
onstration in  order  to  further  illustrate  the  form  of  EASC 
as  a function  of  a for  optimal  (hj^,  h2,  N/a)  . The  actual 
EASC  program  contained  in  Appendix  A does  not  evaluate  these 
points  as  explained  in  step  18  of  Chapter  III.  A modified 
version  of  the  program  was  used  to  obtain  this  data. 

During  a planning  meeting  with  the  instrumentation 
group,  the  fact  was  mentioned  that  a new  control  unit  cost- 
ing $7,000  was  available  for  the  target  drones  which  would 
reduce  variations  in  altitude  by  50  percent.  Commander, 

OTEA  asked  the  test  designer  to  determine  if  this  new  unit 
would  be  worth  the  additional  cost.  In  order  to  evaluate 

this  alternative,  the  test  designer  input  the  new  value  of 

_2 

the  control  variance  for  altitude  , into  the  EASC  program 
maintaining  all  other  parameters  at  their  initial  values. 


I 


The  new  optimal  solution  was  EASC  = $8,897  M,  a reduction  of 
$10,000  from  the  initial  value.  Based  on  this  expected  re- 
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Table  1.  Initial  Input  Data  for  OT  I 


Cost  Coefficients 
(million  dollars) 

Primary 

Parameters 

Cq  = 1.000 

2 

a = 
Y 

4.000 

C = 10.000 
a 

d = 

.200 

C„  = 10.000 

p 

2 

.500 

= .250 

2 

.500 

= .100 

“Iy  ‘ 

. 500 

-2 

0 = 
^2 

2.000 

= 

^3 

2.000 

2 

20.000 

X3 

10.000 

Ik 
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duction  in  EASC,  the  new  unit  was  purchased  and  OT  I was 
conducted . 


Test  Design  for  OT  II 

Based  on  data  collected  during  OT  I,  estimates  of  the 
primary  parameters  are  revised.  and  Cg  are  also  in- 

creased to  reflect  the  expected  increased  expenditure  of 
funds  which  will  occur  between  OT  II  and  OT  III  as  a result 
of  the  ASARC  and  DSARC  recommendations  which  will  be  based 
to  a large  extent  on  the  results  of  OT  II.  has  now  be- 

come larger  than  Cg,  since,  if  the  new  system  is  approved 
for  transition  into  Phase  III,  an  advanced  prototype  will  be 
constructed.  While  if  the  results  of  OT  II  indicate  that 
modifications  to  the  guidance  system  are  necessary,  their 
costs  are  expected  to  be  relatively  minor  and  the  system 
will  be  required  to  undergo  further  limited  testing,  after 
their  completion  and  prior  to  approval  for  transition  into 
Phase  III. 

The  initial  OT  II  input  data  to  the  EASC  program  is 
shown  in  Table  2 and  the  resultant  output  values  are  graphed 
in  Figure  12. 

After  initiation  of  OT  II  testing,  OTEA  is  notified 
by  the  project  manager  that  the  required  performance  margin, 
d,  has  been  reduced  from  .200  to  .150.  Commander,  OTEA 
directs  that  the  current  design  for  OT  II  be  reevaluated  in 
light  of  this  change  and  revised  if  indicated.  Running  the 
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Table  2.  Initial  Input  Data  for  OT  II 


Cost  Coefficients 
(million  dollars) 


C = 1.000 

o 


C = 


c = 


c,  = 


c„  = 


20,000 

IS. 000 

.250 

.100 


Primary  Parameters 


a = 2.500 

Y 


d = .200 


X Y 


X Y 
3 


ZY 


-2 

o 

X, 


-2 

a 

X 


.700 


.600 


.650 


.800 


= 1.400 


'x  " 
^2 


20.000 

10.000 


I 


.'FXPECJtO  ADDED  SYSTEf^i  CO 


Figure  12.  Optimal  (EASC/o)  for  Initial  OT  II  Data. 
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EASC  program  with  the  revised  value  for  d results  in  the 
new  optimal  solution 

a = .21 

N = 18  (T^=8,T2=10) 

B = .2583 
EASC  = $12,074  M 

Decreasing  d results  in  a decrease  in  the  ratio,  r, 

■?  2 

defined  in  Chapter  III  as  d /a^  . Note  the  changes  that 
this  decrease  in  r produces  in  the  optimal  solution.  N de- 
creases from  20  to  18,  a increases  from  .17  to  .21,  and  EASC 
increases  from  $11,299  M to  $12,074  M.  All  of  these  changes 
are  examples  of  the  relationships  described  in  Chapter  III 
and  illustrated  in  Figures  9 and  10.  A decrease  in  d re- 
sults in  an  overall  increase  in  6 for  all  values  of 
(Ti,T2la,N),  which  accounts  for  the  increase  in  the  optimal 
value  for  EASC.  Any  change  in  d will  result  in  an  opposite 
effect  on  the  optimal  value  for  EASC.  It  will  also  result 
in  changes  in  the  optimal  value  of  a in  the  opposite  direc- 
tion but  these  changes  may  not  be  detected  if  the  change  is 
smaller  than  the  increment  between  values  of  a which  are 
evaluated.  Changes  in  the  optimal  value  for  N may  result, 
depending  on  the  effect  that  the  resulting  change  in  r has 
on  AB/&N  for  given  a.  Note  also  that  if  a change  in  N re- 
sults which  produces  a change  in  sampling  cost  large  enough 
to  off-set  the  change  in  EASC  produced  by  d,  the  overall 
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effect  will  be  a reduction  in  EASC.  This  was  not  the  case 
in  this  example. 

As  a result  of  the  new  data,  the  number  of  firings 
for  the  HAWK  missile  are  reduced  from  12  to  10.  In  addition, 
the  value  for  the  significance  level  to  be  used  during  anal- 
ysis is  increased  from  .17  to  .21.  OT  II  is  then  completed. 

Test  Design  for  OT  III 

Estimates  of  the  primary  parameters  are  again  revised 

based  on  the  combined  data  from  OT  I and  II.  C and  C„  are 

a 6 

increased  significantly  since  the  results  of  OT  III  will  be 
used  to  decide  if  the  new  system  should  be  placed  into  pro- 
duction. Again,  however,  is  larger  than  due  to  the 
different  costs  which  are  expected  to  result  from  the  occur- 
rence of  a Type  I error  versus  a Type  II  error. 

The  initial  OT  III  input  data  to  the  EASC  program  is 
shown  in  Table  3.  The  output  values  are  graphed  in  Figure 
13. 

Prior  to  the  start  of  testing,  OTEA  is  informed  by 
the  Army  Material  Command  (AMC)  that  a new  type  of  target 
drone  is  available  which  has  an  improved  speed  control  sys- 
tem. Based  on  the  data  furnished  by  AMC  for  the  new  drone, 
the  test  designer  calculates  that  the  control  variance  for 
speed  could  be  reduced  by  28.5  percent.  The  effect  of  this 

reduction  is  evaluated  using  the  EASC  program.  Inputing  the 

_2 

reduced  value  for  a produces  the  new  optimal  solution 


Table  3.  Initial  Input  Data  for  OT  III 


Cost  Coefficients 
(million  dollars) 


Primary  Parameters 


= 1.000 

= 

2.500 

= 500.000 

d 

= 

.150 

= 150.000 

P ^ 
X^Y 

= 

.600 

= .350 

. 600 

= .100 

p2 

ZY 

= 

. 550 

52 

.800 

X2 

52 

— 

1.400 

X3 

2 

= 

20.000 

= 

10.000 

St6l\ltFICAJ\iCd  LEVEL  (CK) 


Figure  13.  Optimal  (FA.SC/a)  for  Initial  OT  III  Data- 
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a = .05 

N = 32  (T^=12,T2=20) 

6 = .5527 
EASC  = $115,107  M 

The  expected  reduction  in  EASC  is  $.582  M.  The  total  in- 
crease in  cost  to  obtain  the  required  32  new  drones  is 
$.320  M.  Therefore,  the  purchase  of  the  new  drone  is  justi- 
fied by  the  EASC  approach. 

Nothing  that  the  optimal  solution  calls  for  the  max- 
imum number  of  firings  for  both  missile  systems,  the  Com- 
mander, OTEA  asks  the  test  designer  to  evaluate  the  effect 
on  the  optimal  EASC  of  increasing  the  maximum  number  of  HAWK 
firings  by  one.  Inputing  this  change  into  the  EASC  program 
yields  the  new  optimal  solution 

a = .05 

N = 33  (T^=12,T2=21) 

B = .5502 
EASC  = $114,831  M 

Based  on  a comparison  of  the  $.276  M reduction  in  EASC  with 
the  $.100  M cost  of  the  HAWK  missile,  a request  for  one  ad- 
ditional missile  was  submitted. 

Although  the  input  data  used  for  this  example  is  com- 
pletely hypothetical,  it  serves  to  demonstrate  the  applica- 
tion of  the  EASC  approach  to  OT.  The  types  of  decision 
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alternatives  which  may  be  evaluated  using  EASC  are  not 
limited  to  those  contained  in  this  example.  Through  varia- 
tion of  parameters,  numerous  other  alternatives  may  be  eval- 
uated to  include,  for  example,  the  addition  or  deletion  of 
a factor  or  covariate,  estimates  of  the  effects  of  inaccu- 
racies in  estimation  of  primary  parameters,  and  the  effects 
of  variations  of  cost  coefficients. 
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CHAPTER  V 

CONCLUSIONS  AND  RECOMMENDATIONS 

Limitations  of  the  Research 
This  research  is  limited  in  application  to  univariate, 
quantitative,  continuous,  linear  response  models.  Also  it 
is  limited  by  the  additional  assumptions  required  for  ANOVA 
to  the  extent  that  a particular  departure  from  those  assump- 
tions effects  the  validity  of  the  analysis  and  cannot  be 
corrected  for  by  the  application  of  appropriate  techniques. 

The  more  restrictive  assumptions  required  by  ANOCOV  are  only 
applicable  when  this  particular  design  is  actually  employed. 

Another  apparent  limitation  is  the  requirement  that  j 

reasonably  accurate  estimates  of  the  required  input  param-  i 

eters  and  cost  coefficients  be  available.  However,  some  j 

1 

form  of  estimate,  conscious  or  unconscious,  of  these  values  i 

I 

is  inherent  in  any  method  for  selecting  experimental  de-  i 

i 

signs,  although  it  may  not  be  explicitly  stated.  Use  of  the  j 

i 

EASC  approach  forces  the  designer  to  consciously  formulate  j 

these  estimates  and  allows  him,  by  variation  of  parameters,  ! 

to  evaluate  the  sensitivity  of  the  optimal  solution  to  the  1 

< 

( 

accuracy  of  initial  estimates. 

Conclusions 


The  criterion  of  expected  additional  system  cost 


* 
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(EASC)  as  defined  in  this  research  is  a valid  measure  of 
t-he  relative  efficiency  of  alternative  experimental  design 
structures  for  operational  tests,  subject  to  the  limitations 
previously  described. 

The  EASC  approach  represents  a feasible  and  system- 
atic method  for  selecting  specific  experimental  design 
structures  for  operational  testing  under  conditions  of  con- 
strained sample  size. 

As  the  upper  bound  on  sample  size  increases,  this 
approach  quickly  becomes  impractical  due  to  the  rapid  in- 
crease in  the  size  of  the  feasible  solution  set  which  ne- 
gates the  practicality  of  the  partial  enumeration  solution 
procedure  employed. 

Recommendations 

This  research  is  limited  in  application  as  previously 
described.  Operational  testing  generally  involves  multiple 
MOE,  many  of  which  are  either  discrete  or  qualitative.  Ex- 
tension of  this  approach  to  optimization  over  multiple  re- 
sponses or  the  development  of  a multivariate  analog  to  this 
approach  are  suggested  as  subjects  for  future  research.  Ex- 
tension of  this  method  to  discrete  and/or  qualitative  MOE 
is  another  area  in  which  future  research  is  recommended. 

Based  on  an  empirical  analysis  of  the  form  of  the 


objective  function  with  respect  to  the  decision  variables, 
some  form  of  nonlinear  programming  technique  may  be  applica- 


ble  and  more  efficient  for  solving  the  optimization  problem 
as  formulated  in  Chapter  III.  This  is  an  area  in  which 
future  research  might  significantly  extend  the  applicability 
of  this  approach  by  allowing  a much  larger  size  of  feasible 
solution  set  to  be  handled. 

Finally,  it  is  recommended  that  the  US  Army  Opera- 
tional Test  and  Evaluation  Agency  adopt,  on  a trial  basis, 
the  EASC  approach  as  an  adjunct  to  current  selection  proce- 
dures for  situations  which  fall  within  the  limitations  of 
this  approach  in  its  current  infant  state. 
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APPENDIX  A 


This  appendix  contains  a FORTRAN  IV  listing  of  the 
main  program  for  the  EASC  approach  algorithm.  The  program 
receives  input  data  from  a user  created  data  file  which  is 
given  a local  file  name,  TAPES.  The  program  utilizes  two 
subroutines  from  the  CDC  CYBER  70  Math-Science  Library  Pack- 
age to  calculate  the  Type  II  error:  PIFDIS  determines  the 
critical  value  for  the  F test  given  the  value  of  signifi- 
cance level  and  the  numerator  and  denominator  degrees  of 
freedom:  PFDIST  determines  the  value  for  the  Type  II  error 
given  the  adjusted  critical  value  and  the  numerator  and 
adjusted  denominator  degrees  of  freedom  as  indicated  in 
Chapter  II. 

The  user  may  select  three  different  levels  of  output 
by  specifying  the  value  for  PARAM.  PARAM  = 1 will  provide 
a complete  list  of  all  points  evaluated  by  the  program  as 
well  as  identifying  the  optimal  values  for  each  value  of 
significance  level  and  the  overall  optimum.  PARAM  = 2 lists 
all  points  evaluated  and  the  optimum  for  only  one  value  of 
significance  level.  PARAM  = 3 lists  only  the  optimal  points 
for  each  value  of  significance  level  and  the  overall  optimum. 
Appendix  B contains  an  example  of  the  output  format  for  each 
option. 
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PROGRAM  COSTOPT { INPUT , OUTPUT , TAPES , TAPE5=INPUT , TAPE6 ) 

DIMENSION  RHOF(6) 

DIMENSION  RHOX(6) 

DIMENSION  AFVAR(6) ,FVAR(6) 

DIMENSION  NIMIN(IOO) ,N2MIN{100) ,XTCOST(50) 

DIMENSION  TCMIN(IOO) ,SCMIN(100) , BMIN ( 100 ) , BCMIN ( 100 ) 

DIMENSION  XLAMIN(IOO) ,ALAM(50,50) , SCOST ( 50 , 50 ) , VTADJ ( 50 , 50 ) 
DIMENSION  N(2,50) ,XN(2, 50) ,C(2) ,NN(2) 

DIMENSION  MAXNldOO) 

DIMENSION  PARM(2) ,H(2) ,SUMXN(2) 

READ (5,500)  NF ACT , NX , Nl , N2 , ACOF , BCOF , FCOST 

READ (5, 501)  VAR,D 

READ (5, 502)  ALPHA, Cl 

READ (5, 50 3)  RHOF 

READ (5, 503)  RHOX 

READ (5, 504)  AFVAR 

READ (5, 504)  FVAR 

READ (5, 50 5)  C 

READ (5, 506)  IPARAM 

500  FORMAT (213, 214, 2F10.0,F6.0) 

501  FORMAT (F7.0,F8.0) 

502  FORMAT (2F6.0) 

503  FORMAT (6F4 . 0) 

504  FORMAT (6F7 . 0) 

505  FORMAT (2F6.0) 

506  FORf-lAT(12) 

599  FORMAT (*  NUM  OF  FACTORS=* , 13 , / , * NUM  OF  COV=*,I3) 

600  FORMAT  UHl) 

601  FORMAT (IHO,*  RUN  NUMBER-  *,I4,//) 

602  FORMAT (*  PARAMETER  VALUES  ARE-  *) 

603  FORMAT (T2, * ALPHA  COEFF=* ,F10 . 3 ,/,T2 , *BETA  COPFF=* ,F10 . 3) 

604  FORMAT (Tl,* SAMPLE  SIZE  CONSTRAINTS ;*,/ ,T26 , *TESTED  ITEM=* , 
+ 13) 

6041  FORMAT (T2 6, +STANDARD  ITEM=*,I4) 

605  FORMAT (T 2,* VAR  FOR  RANDOM  DESIGN=* , F7 . 2 ) 

606  FORMAT (T 2, *REQ  PERFORMANCE  MARGIN=* , F7 . 2 ) 

607  FORMAT (T2,*CORRELATION  COEFFICIENTS:*) 

608  FORMAT (T27,*FACTOR  * , 13 , *=* , F4 . 2 ) 

609  FORMAT (T2 7, *COVAR  * , 13 , *=* ,F4 . 2) 

610  FORMAT (T12, *AVG  VAR  ABOUT  FIXED  LEVEL* , T4 8 , *POP  VAR*) 

611  FORMAT(T2,*FACTOR*,I3,T21,F7.3,T51,F7.3) 

612  FORMAT (T2,*DESIGN  ERROR  VAR=* ,F10 . 4 , /) 

613  FORMAT (T2,*SAMPLE  COSTS :* ,/,Tl6 , *TESTED  ITEM=  *,F6.3) 

614  FORMAT  (Tl  6,  *STAfJDARD  ITEM=  *,F6.3) 

700  FORMAT (///,*  ALPHA=  *,F6.4) 

701  FORMAT (*  ALPHA  COST=  *,F10.3) 

702  FORMAT(lHO,T2, *NO  OBS* , /, T2 , *ROWl* , 17 , *ROW2* , T14 , *SCOST* , 
+T22 , *LAMBDA* ,T31 , *BETA* ,T39 , *BCOST* ,T55 , *TCOST* , /) 

704  FORMAT(T2,I3,I7,I3,T14,F7.3,T22,F7.3,T31,F6.4,T39,F10. 3, 
+T55,F11.3) 

708  FORMAT(//,*  FOR  ALPHA=  *,F6.4,*  OPTIMAL  VALUES  ARE:*,/) 
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710  FORMAT (//,*  OVERALL  OPTIMAL  ALPHA=  *,F6.4) 

711  FORMAT (*  MINIMUM  TOTAL  COST=  *,F11.3) 

712  FORMAT (//,*  END  OF  RUN  NUMBER:  *,I3) 

722  FORMAT (2H0,*  BEGIN  OUTPUT  DATA*) 

723  F0RMAT(T2,I2) 

NRUN=0 

IZ=2 

IPROD=(IZ) ** (NFACT) 

LIMCN=(IZ) ** (NFACT-1) 

YLIMCN=FLOAT (LIMCN) 

NRUN=NRUN+1 

WRITE(6,600) 

WRITE (6, 601)  NRUN 
WRITE (6, 599)  NFACT, NX 


WRITE (6, 602) 

WRITE (6, 60 3)  ACOF,BCOF 
V7RITE  (6 , 613)  C(l) 

WRITE(6,614)  C(2) 

WRITE (6, 604)  N1 
WRITE (6 . 6041)  N2 
WRITE (6, 605)  VAR 
WRITE (6, 606)  D 
WRITE (6 ,607) 

199  DO  299  1=2, NFACT 

299  WRITE (6, 608)  I,  RHOF(I) 

198  DO  298  1=1, NX 

298  WRITE(6,609)  I,  RHOX(I) 

WRITE (6 , 610) 

197  DO  297  1=2, NFACT 

297  WRITE(6,611)  I,  AFVAR(I),  FVAR(I) 

BIGM=(10.0) ** (6.0) 

GTC=BIGM 

MINVE=IPROD=NX 

MAXVE= (N1+N2) - (IPROD+NX) 

EVAR=VAR 

201  DO  301  1=2, NFACT 

301  EVAR=EVAR* ( 1 . 0- (RHOF ( I ) * ( 1 . 0- (AFVAR ( I) /FVAR ( I ) ) ) ) ) 

202  DO  302  1=1, NX 

302  EVAR=EVAR* (l.O-RHOX(I) ) 

WRITE (6, 612)  EVAR 

WRITE (6 , 722) 

ICOUNT=l 
999  CONTINUE 
P=l. 0-ALPHA 
IF(P)  1000,1000,11 
11  ACOST=ACOF* ALPHA 
WRITE (6, 700)  ALPHA 
WRITE (6, 701)  ACOST 
COFFL= ( D*  * 2 ) /EVAR 


203  DO  303  K=l,2 
2031  DO  303  J=2,LIMCN 
303  N(K,J)=2 

IFdPARAM  .EQ.  3]  GO  TO  46 
WRITE (6, 702) 

46  TCA=BIGM 

888  DO  800  I=MINVE,MAXVE 

TCLOW=BIGM 

PARM(2)=FLOAT(I) 

NOBS=I+IPROD+NX 
IFdPARAM  .EQ.  3)  GO  TO  45 
WRITE (6, 723)  NOBS 
45  NDIF=NOBS- dPROD+N2) 

NEQO=NOBS/2 
NMl=IPROD 
N1LIM=MAXN1 (NOBS) 

IF(N1LIM  .GE.  NMl)  GO  TO  19 
N1LIM=N1 

19  IF(NDIF)  21,21,20 

20  NM1=NM1+NDIF 

21  NMLIM=MINO(NEQO,Nl,NlLIM) 

899  DO  900  IJ=NM1,NMLIM 

NN (1) =IJ 

NN (2) =NOBS-NN (1) 

IFdCOUNT  .GE.  2)  GO  TO  805 

SCOST(NNd)  ,NN(2)  ) = (NN(1)  *C  (1 ) ) + (NN  ( 2 ) *C  ( 2 ) ) 
207  DO  307  K=l,2 
2071  DO  3071  J=1,LIMCN 
3071  N(K,J)=NN(K)/LIMCN 


IMOD=MOD(NN(K) ,LIMCN) 

IF(IMOD)  307,307,208 
208  DO  308  J=l,IMOD 
308  N(K,J)=N(K, J)+l 
307  CONTINUE 
SUMXM(1)=0.0 
SUMXN(2)=0.0 
205  DO  305  K=l,2 
DO  3051  J=1,LIMCN 
XN(K,J)=FLOAT(N(K,J)  ) 

3051  SUMXN(K)=SUMXN(K)  + (1.0/.XN(K,J)  ) 

305  H(K)=YLIMCN/SUMXN(K) 

ALAM(NN (1) ,NN (2) )=( (COFFL*H(l) *H ( 2 ) ) / (H ( 1 ) +H ( 2 ) ) ) 
VTADJ(NN(1) ,NN(2) ) = ( 1 . 0+ALAM (NN ( 1 ) , NN ( 2 ) ) ) * * 2/ ( 1 . 0+ ( 2 . 0 
+*ALAM(NN(1) ,NN(2) ) ) ) 

805  CONTINUE 
PARM(1)=1.0 
FC=PIFDIS (P,PARM,IR) 

X=FC/(1.0+ALAM(NN(1) ,NN(2) ) ) 

PARM ( 1 ) =VTADJ (NN ( 1 ) , NN ( 2 ) ) 
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BETA=PFDIST (X,PARM) 

BCOST=BCOF*BETA 

TCOST=FCOST+SCOST (NN (1) ,NN(2) ) +ACOST+BCOST 
IFdPARAM  .EQ.  3)  GO  TO  44 

WRITE(6,704)  NN(1) ,NN(2) ,SCOST(NN(l) ,NN(2) ) ,ALAM{NN(1) ,NN{2) ) , 
+BETA , BCOST , TCOST 
44  XTCOST (IJ) =TCOST 
IF (IJ- (NMl+1) ) 24,24,23 

23  TCl=XTCOST(IJ-2) 

TC2=XTCOST (IJ-1) 

TC3=XTCOST (IJ) 

IF(TC2  .LE.  TCI  .AND.  TC2  . LE . TC3)  GO  TO  901 

24  IF  (TCOST  .GE.  TCLOV7)  GO  TO  900 
SCMIN(NOBS)=SCOST(NN(l) ,NN(2) ) 

XLAMIN (NOBS) = ALAM (NN ( 1 ) , NN ( 2 ) ) 

TCMIN (NOBS) =TCOST 

BMIN (NOBS) =BETA 
BCMIN (NOBS) =BCOST 
MlMIN (NOBS) =KN (1) 

N2MIN (NOBS)=NN (2) 

TCLOW^TCMIN (NOBS) 

900  CONTINUE 
GO  TO  902 

901  MAXNl(NOBS)=IJ-l 

902  ATC=TCMIN(NOBS) 

*IF(TCA-ATC)  801,800,25 

801  AC1=TCMIN (NOBS-2) 

AC2=TCMIN(NOBS-l) 

AC3=TCMIN (NOBS) 

IF(AC2  .LE.  ACl  .AND.  AC2  . LE . AC3)  GO  TO  802 

25  TCA=ATC 
N1A=N1MIN (NOBS) 

N2A=N2MIN (NOBS) 

NTA=NOBS 

SCA=SCMIN(NOBS) 

BA=BMIN (NOBS) 

BCA=BCMIN(NOBS) 

XLA=XLAMIN (NOBS) 

800  CONTINUE 

802  IFdPARAM  .EQ.  3)  GO  TO  47 
VmiTE (6 , 708)  ALPHA 

47  WRITE(6,702) 

WRITE (6,723)  NTA 

WRITE (6,704)  NlA,N2A, SCA , XLA , BA , BCA , TCA 
IFdPARAM  .EQ.  2)  GO  TO  1005 
**IF (GTC-TCA)  1000,50,26 

26  GTC=TCA 


J 


GALPHA=ALPHA 
50  ALPHA=ALPHA+CI 


MICROCOPY  RESOLUTION  TEST  CH«RT 

NATIONAL  BURtAU  OF  STANDARDS  l963-^t 
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IC0UNT=IC0UNT+1 
GO  TO  999 
1000  CONTINUE 
WRITE (6, 7 10)  GALPHA 
WRITE (6,711)  GTC 
WRITE (6, 712)  NRUN 
1005  STOP 
END 


*Changing  801  to  800  in  this  statement  will  cause  the  program 
to  evaluate  and  print  the  appropriate  output  for  all  values 
of  (N/ci)  rather  than  stopping  one  increment  beyond  the  local 
minimum  (n^^,  n2,  N/a)  . 

**Changing  1000  to  50  in  this  statement  will  cause  the  program 
to  evaluate  and  print  the^  appropriate  output  for  all  values 
of  a rather  than  stopping*  one  increment  beyond  the  optimal 
n2 » a)  value.  |i 


APPENDIX  B 


This  appendix  contains  an  example  illustrating  the 
output  options  which  are  available  from  the  EASC  program, 
COSTOPT,  by  specifying  the  value  of  the  input  parameter, 
PARAM,  as  described  in  Appendix  A.  Unless  otherwise  indi- 
cated on  the  figures  themselves,  the  input  data  listed  for 
this  example  was  used  to  obtain  the  data  for  all  of  the 
figures  contained  in  Chapter  III,  with  the  exception  of 
Figure  4. 
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N'JM  or  FAOTO»S=  5 
NUM  or  COV=  i 
PflRAMETFP.  VatUEF  a-’L- 
flLPHA  CCF:rF=  . jQo 

BFTA  COFFFs  70.0CO 
SAF'f’I.F  COSTS! 

irSIr'  IT-'M=  1.000 
STA^na-Jn  it£M=  .500 
SAF'^Lt  SI7£  CONCT->Al‘JTS! 

Tc.sjl::  ir^Mr  12 
STaNOARO  n~M=  ?o 
VAP  FOR  RANUO"  CfSIOki:  2.00 

DFQ  prRFo-^'iaNCE  fn-’oiK=  .13 
correlation  COfFFIGIENTSt 

FACTOR  2=  .70 
FACTOR  3=  .60 
COVAR  1=  .70 

avo  VAR  AOOMT  FlXrC  L'VfL  °0“  VA^ 


FACTOR  2 
FACTOR  3 
DESIGN  ERROR  VA^r 


2.000 

2.noo 

.1154 


20.000 
1 c . 0 e 0 

(This  value  is  calculated  within 
the  proprem) 


, BEGIN  OUTPUT  '^ATA 


ALPHa=  .0100 

ALPHA  C05T=  5.000 

NO  OOS 

POWl  KOW2  SCOST  LAM'^OA  R£  TA  PCCST  TCOST 


16 


0 

17 

8 

12.000 

3.347 

. 87  81 

6 1 * 4 o ^ 

79.^69 

8 

R 

12.503 

3.362 

. 87,07 

hij  9 ^06 

79.306 

18 

B 

10 

l'.«.  00  3 

3.378 

. “6  0 5 

60.234 

7 R . 2 3 4 

R 

R 

1 3 • j C 

3.378 

. 86^5 

60.234 

79. 7M* 

IR 

8 

11 

13.500 

3.3R6 

. 8533 

5 R . 7 3 1 

7R.231 

R 

10 

14.000 

3.306 

.85’3 

5 3.731 

7R.731 

20 

6 

12 

14.000 

3.416 

. “4r4 

5 R . 2 8 1 

7R.281 

R 

11 

14.500 

3.416 

. 846R 

5R. 281 

7R.7  81 

10 

10 

1 5 . [ 2 0 

3.416 

.8^65 

5'3.2  31 

80.281 

21 

88 


8 

13 

1 *♦  . 5 0 0 

3.425 

. 8.*lf 

58.9*4 

79.414 

9 

1? 

1 5 . c '1  a 

’ .4  >8 

. ^>.ir 

58.a’0 

79. 870 

10 

11 

15.510 

3.433 

. “-.ir 

58.870 

80.370 

22 

8 

1!4 

i3.c:  j 

3.43  8 

. o-5&q 

58.585 

79.585 

9 

13 

15.500 

3.4  30 

. “362 

58.537 

80.037 

to 

1 2 

15.010 

3.462 

. 8355 

58.487 

eC  .4«7 

11 

11 

15.510 

^.462 

. “35'-. 

58.487 

en. )»’ 

23 

fl 

15 

15.500 

3.450 

.8127 

58.286 

79.780 

9 

!(< 

1 -0  . 0 0 C 

’.46? 

. ^319 

58.235 

80.235 

10 

13 

15.500 

3.475 

.8311 

58.180 

80.5  80 

11 

12 

17.000 

3.489 

. ■»  3 C 3 

5 8.123 

81.123 

24 

8 

16 

16. COO 

3.h52 

. 8287 

58.011 

80.011 

9 

15 

15.500 

3.-75 

. 8?’9 

57.956 

8 C . 4 5 6 

10 

14 

17.000 

3.489 

. ’271 

57.807 

80.897 

11 

13 

17.500 

3.504 

. “262 

5^.355 

81 .335 

12 

12 

18.000 

3.520 

. 8253 

57.769 

81.76° 

25 

8 

17 

16.500 

3.470 

. 82  54 

57.780 

80 .280 

9 

16 

17.000 

3.489 

. 0242 

57.697 

80.697 

10 

15 

17.500 

3.504 

. ’2  33 

57.633 

“1.133 

11 

14 

18.CjO 

3.520 

. 8224 

57.566 

"1 . 566 

12 

13 

18.500 

3.537 

. 821  3 

57.494 

81 . 994 

26 

R 

18 

17.000 

3.473 

. 8224 

57 . 566 

“0.566 

9 

1 7 

17.503 

3.498 

. 8211 

57.479 

3" . 979 

10 

16 

18.000 

3. 52C 

. £1°8 

57.  '85 

31 .3“5 

11 

15 

18.500 

3.537 

. «1  «7 

57.311 

81  . “11 

12 

14 

11. COD 

3.554 

. 8176 

57.233 

“2.23’ 

27 

8 

19 

17.500 

3.486 

. “196, 

57.369 

80 . “6° 

9 

18 

18.000 

3.5C7 

. ><1  “? 

57.277 

81.277 

in 

17 

18.500 

’.530 

. “168 

57.177 

“1.67’ 

11 

16 

19.000 

3.554 

. “15’ 

57.068 

82.36“ 

12 

15 

11.5  00 

3.574 

. “1  40 

56.983 

32.-“’ 

28 

8 

20 

1 8 . 0 J C 

3.495 

.8169 

57.185 

8 1 . 1 8 5 

9 

19 

18.500 

3.517 

. 8155 

57.038 

81.588 

10 

1» 

19. COO 

3.540 

. “147 

56.98’ 

ei . J“’ 

11 

17 

19. 5') 0 

3.56  6 

. 8124 

56. 8o8 

82.36“ 

12 

16 

2 0 . C 0 D 

3.594 

. 8 1 : 6 

56. 741 

82. 741 

29 

9 

?0 

19.000 

3.526 

. 81  30 

56.910 

8 3 . n 0 

10 

n 

19.500 

3.551 

.8114 

56.800 

82.7CC 

11 

18 

’0 . C 0 j 

’ . 578 

. “097 

56.678 

•2.578 

12 

17 

70 .300 

3.607 

.“'078 

56.5-4 

33.044 

30 

r 


10 

20 

■»0.  C :J  J 3.062 

. •'0  ■»9 

56.625 

A2.625 

11 

19 

0 0.0,10  3.590 

.0071 

56.417 

F2. 997 

12 

10 

21.000  3.621 

. oj^l 

56.355 

53.355 

31 

11 

20 

21.000  3.o0.'’ 

. O'lAe 

56.322 

53.322 

12 

19 

21.500  3.635 

. ‘•:?5 

56.173 

83.673 

32 

12  20 

FOR  AL"KA= 

NO  ORS 

22.000  3.650 

• OICO  C)3ri*'AL 

. 7 9;  9 

VALU3S  A 

55.995 

1 

83.905 

ROWl 

90W2 

SCOST 

5t.  TA 

ecnsT 

rcosT 

19 

A 

11 

13.500  ■'.396 

. “5  33 

59.731 

79.221 

The  above  represents  the  output  format  for  PARAM  = 2 
PARAM  = 1 produces  the  same  output  as  above  for  all  values  of 
significance  level  considered,  and  in  addition  includes  the 
following  at  the  end  of  each  run: 


OVfi^AUL  OPTIMAL  AL=>HA=  .0100  ■ 
MIMMIJ'I  TorAL  CCST=  79.^31 


F.NC  OP  RUN  NUf-ofRI  1 


PARAM  = 3 produces  only  the  optimal  value  for  each 
value  of  significance  level  considered  and  the  overall  optimum. 
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